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1. Introduction 

Recently, a new approach to functoriality of automorphic representations [Llj has 
been proposed in [FLN] following earlier work \L2\ \L3\ IL4] by Robert Langlands. The 
idea may be roughly summarized as follows. Let G and H be two reductive algebraic 
groups over a global field F (which is either a number field or a function field; that is, 
the field of functions on a smooth projective curve X over a finite field), and assume 
that G is quasi-split. Let ^G and be the Langlands dual groups as defined in 
|L1| . The functoriality principle states that for each homomorphism — >■ ^G there 
exists a transfer of automorphic representations from H{A) to G{A), where A is the 
ring of adeles of F. In other words, to each L-packet of automorphic representations 
of H(A) should correspond an L-packet of automorphic representations of G(A), and 
this correspondence should satisfy some natural properties. Functoriality has been 
established in some cases, but is unknown in general. 

In \L2\ IL31 IL41 IFLNj the following strategy for proving functoriality was proposed. 
In the space of automorphic functions on G{F)\G{A) one should construct a family of 
integral operators which project onto those representations which come by functoriality 
from other groups H. One should then use the trace formula to decompose the traces 
of these operators as sums over these H (there should be sufficiently many operators 
to enable us to separate the different H). The hope is that analyzing the orbital side 
of the trace formula and comparing the corresponding orbital integrals for G and H, 
one should ultimately be able prove functoriality. 

In the present paper we make the first steps in developing geometric methods for 
analyzing these orbital integrals in the case of the function field of a curve X over a 
finite field Fg. We also suggest a conjectural framework of "geometric trace formulas" 
in the case of curves defined over the complex field. 

In |FLNj a family of averaging operators ^d,p has been constructed. They depend 
on a positive integer d and an irreducible representation p of ^G. The operator ^d,p is 
expected to have the following property: for sufficiently large d, the representations of 
G(A) that contribute to the trace of p are those coming by functoriality from the 
groups H such that the dual group has non-zero invariant vectors in p. To prove 
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this, we wish to use the trace formula 

(1.1) TrKd,p = J Kd,p{x,x) dx, 

where Kd,pix, y) is the integral kernel of Krf,p on G{F)\G{A) x G{F)\G{K)E Thus, we 
need to express the right hand side of this formula as the sum of orbital integrals and 
to compare these orbital integrals for G and the groups H described aboveH 

Here an important remark is in order. There are two types of automorphic repre- 
sentations of G{A): tempered (or Ramanujan) and non-tempered (or non-Ramanujan), 
see Section [3.31 for more details. As explained in |FLN] . for the above strategy to work, 
one first needs to remove the contribution of the non-tempered representations from 
the trace of ^d,p- This was the focus of [FLNj . where it was shown how to separate 
the contribution of the trivial representation of G(A) (which is in some sense the most 
non-tempered). 

In the first part of this paper we show that the kernel K^^p may be obtained using the 
Grothendieck's faisceaux-fonctions dictionary from a perverse sheaf JCd,p on a certain 
algebraic stack over the square of Bunc, the moduli stack of G-bundles on X. Hence the 
right hand side of the trace formula (jl.ip may be written as the trace of the Frobenius 
(of the Galois group of the finite field k, over which our curve X is defined) on the etale 
cohomology of the restriction of 'Xd,p to the diagonal in Bunc x Buufj. We show that 
the latter is closely related to the Hitchin moduli stacks of Higgs bundles on X. The 
idea is then to use the geometry of these moduli stacks to prove the desired identities of 
orbital integrals for G and H by establishing isomorphisms between the corresponding 
cohomologies. In Section |4] we formulate a precise conjecture relating the cohomologies 
in the case that G = SL2 and H is a twisted torus. 

A prototype for this is the proof of the fundamental lemma (in the setting of Lie 
algebras) given by one of the authors |N3] . However, our situation is different. The 
argument of |N3| used decomposition of the cohomology of the fibers of the Hitchin 
map under the action of finite groups, whereas in our case the decomposition of the 
cohomology we are looking for does not seem to be due to an action of a group. 

One advantage of the geometric approach is that the moduli stacks and the sheaves 
on them that appear in this picture have natural analogues when the curve X is defined 
over k = C, rather than a finite field. Some of our questions may be reformulated over 
C as well, and we can use methods of complex algebraic geometry (some of which have 
no obvious analogues over a finite field) to tackle them. 



More precisely, we should consider the stabilized version of the trace formula, which is more efficient 
for comparing different groups. 

We note that in general we should insert additional operators at the finite set S of closed points 
in X. However, in this paper, in order to simplify the initial discussion, we will restrict ourselves to 
the case when S is either empty or the operators inserted belong to the spherical Hecke algebra (with 
respect to a particular choice of a maximal compact subgroup of G(A)). Thus, only the unramified 
representations of G{A) (with respect to this subgroup) will contribute to (|l.ip . 
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This leads to a natural question: what is the analogue of the trace formula (ll.ip over 
C? We discuss this question in the second, more speculative, part of the paper and 
formulate two conjectures in this direction. 

The geometric analogue of the right hand (orbital) side of (jl.ip is the cohomology 
of a sheaf on a moduli stack, as we explain in the first part. We would like to find a 
similar interpretation of the left hand (spectral) side of (jl.ip . Again, to simplify our 
discussion, consider the unramified case, but with a more general kernel K instead of 
K^ p which is bi-invariant with respect to a fixed maximal compact subgroup G{0) of 
G(A). Then the left hand side of (jl.ip may be rewritten as follows (we ignore the 
continuous spectrum for the moment): 

(1.2) J^m^iV^, 

71" 

where the sum is over irreducible representations vr of G{A), unramified with respect 
to G(0), TTT-TT is the multiplicity of vr in the space of automorphic functions, and N.,^ is 
the eigenvalue of K^i p on a spherical vector in vr fixed by this subgroup. We wish to 
interpret this sum as the Lefschetz trace formula for the trace of the Frobenius on the 
etale cohomology of an ^-adic sheaf. It is not obvious how to do this, because the set 
of the vr's appearing in (jl.2p is not the set of points of a moduli space (or stack) in any 
obvious way. 

However, recall that according to the Langlands correspondence f_LlJ, the L-packets 
of irreducible (unramified) tempered automorphic representations of G(A) are supposed 
to be parametrized by (unramified) homomorphisms 

where Wp is the Weil group of the function field F and is the Langlands dual group 
to G. This has been proved by V. Drinfeld for G = GL2 |DH ID2j and L. Lafforgue for 
G = GLn,n > 2 |LLaf| . but is unknown in general. Nevertheless, assume that this is 
true. Then the eigenvalues are determined by the Hecke eigenvalues of tt, and in 
particular they are the same for all vr in the L-packet Lq- corresponding to any given 
a : Wp — ^G. We will denote them by No-. Further, we expect that if a is unramified, 
then there is a unique representation vr in the L-packet Lq- which is unramified with 
respect to the subgroup G(0). Hence we will write rria for mjr- Then, assuming the 
Langlands correspondence, we obtain that (|1.2p is equal to 

(1.3) ^m^N,. 

a 

Ideally, we would like to describe (II. 3p as the trace of the Frobenius on the etale 
cohomology of a sheaf on a moduli stack, whose set of fc-points is the set of equivalence 
classes of homomorphisms a. Such a stack does not exist if the curve X is defined 
over a finite field k. But when X is defined over C, we have an algebraic stack Loclq 
of de Rham local systems on X. Hence we can pose the following question: define a 
sheaf on this stack such that its cohomology (representing the left hand side of (|l.ip in 
the complex case) is isomorphic to the cohomology representing the right hand side of 
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(jl.ip . It would then be natural to call this isomorphism a geometrization of the trace 
formula (jl.ip . 

Our main observation is that the answer may be obtained in the framework of a cat- 
egorical form of the geometric Langlands correspondence, which is a conjectural equiv- 
alence between derived categories of coherent sheaves on the moduli stack Locl^j of 
^G-local systems on a complex curve X and D-modules on the moduli stack Bunc of 
G-bundles on X. Such an equivalence has been proved in the abelian case by L. Laumon 
|Laul| and M. Rothstein [R], and in the non-abelian case it has been suggested as a 
conjectural guiding principle by A. Beilinson and V. Drinfeld (see, e.g., [Fll IVLafl ILL| 
for an exposition) □ 

So far, this correspondence has been mostly studied at the level of objects. For ex- 
ample, the skyscraper sheaf supported at a given ^G-local system E on X should go 
to the Hecke eigensheaf on Bunc with "eigenvalue" E. But if we have an equivalence 
of categories, then we also obtain non-trivial information about morphisms; namely, 
the Horn's between the objects corresponding to each other on the two sides should be 
isomorphic. The main point of the second part of this paper is that for suitable ob- 
jects the isomorphism of Hom's yields the sought-after geometric trace formula. (More 
precisely, these objects are sheaves on the squares of Bunc and Locl^, which may be 
interpreted as "kernels" of functors acting on the above two categories.) 

We propose a conjectural geometrization of the trace formula (jl.ip in this framework. 
This is still a tentative answer, because several important issues remain unresolved, 
as we explain in Section 16. 5i Nevertheless, we believe that it contains interesting 
features and hence even in this rough form it might provide a useful framework for a 
better geometric understanding of the trace formula as well as the geometric Langlands 
correspondence. 

In deriving this formula, we apply the categorical form of the geometric Langlands 
correspondence to the sheaf 3C, which is the geometric incarnation of the kernel of the 
integral operator K. This sheaf is defined on an algebraic stack over the square of Bunc. 
It is natural to ask whether we can obtain meaningful analogues of the trace formula 
by applying the categorical Langlands correspondence to sheaves on Bunc itself. 

In Section [7] we show that there is in fact an analogous statement which may be 
interpreted as a geometric analogue of the "relative trace formula" (also known as 
Kuznetsov trace formula, see, e.g., [J]). This formula has some interesting features. 
First, since it involves Whittaker functionals, only generic automorphic representations 
appear on the left hand side and hence, conjecturally, the non-tempered representations 
should not appear at all. Second, conjecturally, the factor in the sum (|1.3p should 
disappear because each generic L-packet is expected to contain a single irreducible 
representation with a non-zero Fourier coefficient. 

The price we pay for this is that in the sum (jl.3p appears a weighting factor, 
L((T, ad, 1)~^, the reciprocal of the value of the L-function of a in the adjoint represen- 
tation at s = 1 (here we assume that G is simple, but our analysis may be extended to 



This categorical version of the geometric Langlands correspondence also appears naturally in the 
S'-duality picture developed in [KWj (see [F2| for an exposition) . 
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more general reductive groups). Thus (not counting automorphisms of a), we obtain 
the sum 



Conjecturahy, for non-generic L-packets the L-function L{a, ad, s) has a pole at s = 1 
(see \GP\ El llclkj ) and hence these representations should disappear from the sum 



The insertion of this factor in the context of the trace formulas discussed in [ L2j 
(which are very close to the trace formulas considered here) was originally suggested 
by P. Sarnak [S] and further studied by A. Venkatesh [Vej . for the group GL2 in the 
number field context. From our point of view, this factor has a natural geometric 
interpretation, as one coming from the Atiyah-Bott-Lefschetz fixed point formula (see 
Sections 15.21 16.41 and 17.41 for more details). 

On the right hand side of the relative trace formula we obtain certain analogues of 
the Kloosterman sums, which are represented geometrically by cohomologies of moduli 
stacks that are similar to the ones appearing in the geometrization of the ordinary trace 
formula and have been previously studied in |N1 ] and [FGVl]. Understanding relations 
between these Kloosterman sums, and the corresponding cohomologies, for G and the 
groups H discussed above may give us another way to approach functoriality. 

The paper is organized as follows. In Section [2] we introduce the main ingredient of 
the geometric trace formula, the sheaf 'X.d,p on the square of Bunc and its analogues. 
This sheaf, viewed as a kernel, gives rise to a functor acting on the derived category 
of sheaves on Bunc. In Section [3] we consider the action of this functor on Hecke 
eigensheaves. We prove that for large d this functor annihilates those eigensheaves 
which do not come from smaller groups H by functoriality. Next, we look at the right 
hand (orbital) side of the trace formula in Section [H We show that it corresponds to 
the cohomology of the restriction of our sheaf to the diagonal. We study this restriction 
in more detail and connect it to Hitchin type moduli stacks of Higgs bundles. We then 
formulate a precise conjecture linking the cohomology in the case that G = SL2 and H 
is a twisted torus. 

In the second part of the paper, we begin by describing a conjectural geometrization 
of the left hand (spectral) side of (jl.ip Section \5\ Then we explain in Section [6] how a 
geometrization of the trace formula appears in the context of the categorical Langlands 
correspondence. We state the geometric trace formula in Conjecture 16.11 Finally, we 
develop the needed formalism and state the relative geometric trace formula (Conjec- 
ture I7.2P in Section [71 As a byproduct of this discussion, we formulate a conjectural 
generalization of the Atiyah-Bott-Lefschetz fixed point formula for algebraic stacks in 
Section 16.41 (see Conjecture 16. 2p . 
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Part I 



2. The right hand side of the trace formula 

Let X he a smooth projective geometrically connected curve X over a field k which 
will be a finite field Fg or the field of complex numbers. Denote by F the field of 
rational functions on X and A the ring of adeles of F. For each closed point x G |Ar| 
we denote by Fx the completion of F at x and by 0^^ its ring of integers. 

Let G be a reductive group scheme over X which is a quasi-split form of a split 
connected reductive algebraic group G over the field k. This means that G becomes 
split after pull-back to an etale cover X' of X. 

We wish to find a geometric incarnation of the trace formula 



(2.1) TrK= J K{x,x)dx, 

GiF)\GiA) 

where K is the operator on the space of automorphic functions acting by the formula 



(K./)(x)= I f{xy)H{y)dy, 



G{A) 

where H isa smooth compactly supported function on G(A). We choose a Haar measure 
on G{A) normalized so that the volume of the fixed maximal compact subgroup 

G(0) = n GiOx) 

x€\X\ 

is equal to 1. 

We can rewrite K as an integral operator corresponding to the kernel 

(2.2) K{x,y)= H{x-'ay), 

a(iG{F) 



acting as follows: 



(K-/)(x)= j K{x,y)f{y)dy. 



G{F)\G{A) 

In what follows we will restrict ourselves to unramified automorphic representations 
and we will assume that K is in the (restricted) tensor product of the spherical Hecke 
algebras (with respect to G{Ox)) over all closed points x G \X\. 
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Let Bunc be the algebraic moduli stack of G-bundles on X. Denote by ker^(F, G) 
the kernel of the map 

xe\x\ 

For simplicity, we will assume that ker^(F,G) = 0. Then we have 

Buncik) = G{F)\G{A)/G{0), 

where BunG(A;) is the set (more precisely, groupoid) of /c-points of the moduli stack 
Bunc of G-bundles on X. Hence the kernel K may be viewed as a function on 
BunG(A;) x BunG(A;). 

Then the right hand side of (jl.ip may be rewritten as 

(see, e.g., |BeD hj ) . For a class of kernels K described below, we will give a geometric 
interpretation of this sum as the trace of the Frobenius on the cohomology of an ^-adic 
sheaf on an algebraic stack over the diagonal in Bun^ x Buncr. 

2.1. The dual group. Let ^G be the Langlands dual group to G, 

^G = Gxr. 

Here G is the dual group of G taken here to be defined over the field of complex numbers 
if k = C but to be over (where i is relatively prime to q) if k = ¥q, and F is the 
(etale) fundamental group tti{X,x) of X for some geometric point x ol X that acts 
on G through a finite quotient. This action determines the quasi-split form G. In 
particular, if this action is trivial, then ^G = G x F and G = X x G. 

We will fix a splitting of G and G so that in particular we can talk about dominant 
(co)weights. The fundamental group F acts on the abelian group X* of all weights 
(or characters) of the maximal torus of G, preserving the cone X+ consisting of the 
dominant ones. Recall that X^- is equipped with a partial order: /i > /i' if and only if 
/i — ^' is a sum of simple roots of G. Since the action of F preserves the set of simple 
roots, it also preserves the partial order on X+. It also induces a partial order on the 
set [X+/F] of orbits of F on X+. 

2.2. The Hecke stack. Recall that if we have two G-bundles E, E' on the (formal) 
disc Spec which are identified over the punctured disc Spec fc((t)), we obtain a point 
in the double quotient 

G[[t]]\G((t))/G[M], 
or, equivalently, a G[[i]]-orbit in the affine Grassmannian 

Gr = G((i))/G[[t]] 

which is an ind-scheme over k |BD1 IMVj . These orbits are called Schubert cells, and 
they are labeled by elements ^ the set X-|- of dominant wights of the maximal torus 
in the dual group G. We denote the orbit corresponding to /i by Gr^. We will write 
\nY[E,E') = if the pair {E,E') belongs to Gr^. Recall that Gr^/ is contained in the 
closure Gr^ of Gr^ if and only if /x > 
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Recall the Hecke stack !K = "K (X, G) (see [BP] ) that classifies quadruples 

where x £ X, E,E' & Bun^ and (p is an isomorphism 

E\x~{x} — E'\x--{x}- 

We have two natural morphisms p,p' : "K ^ Bun^ sending such a quadruple to E or E' 
and the morphism s : 'K ^ X. Since Bun^j is an algebraic stack, so is J{(X, G). How- 
ever, if we fix E' , then we obtain an ind-scheme over X, which is called the Beilinson- 
Drinfeld Grassmannian (see [BD^ IMVj ) . 

Let 'X'{X,G) be the stack classifying the quadruples 

ix,E,E',<P), 

where x £ X , E £ Bun^, E' is a G-bundle on the disc around the point x, and (p 
is an isomorphism 

E\d'^:^E'\d'^, 

where D* is the punctured disc around x. We have a natural morphism 

J{{X,G) 5{'(X,G) 

(restricting E' to Dx and (j) to D'), which is in fact an isomorphism, according to 
a strong version of a theorem of Beauville-Laszlo [BL] given in [BD], Sect. 2.3.7. 
Therefore we obtain that the morphism 

sxp: :K{X,G) ^X X BuuG 

sending the above quadruple to (x, E) is a locally trivial fibration with fibers isomorphic 
to the affine Grassmannian Gr = G((t))/G[[t]]. 

For every orbit ^ E X^. we define the closed substack Jf^ of 5{(X, G) by imposing 
the inequality 

(2.4) inv^(^,^') < ^• 

It is a scheme over X x Bun^j with fibers isomorphic to Gr^. 

Suppose now that G is a quasi-split form of G whose dual group ■^G is a semi-direct 
product G X r given by a finite action of T on G, fixing a given splitting. Let T' be the 
kernel of this action which is a normal subgroup of finite index in V. This is equivalent 
to a finite etale covering vr : X' — )• X with a geometric point x' G X' over x £ X such 
that the pull-back of G to X' is split. The group scheme G over X is isomorphic to 

X' X G, 
r/r' 

and a G-torsor on X is nothing but a G-torsor E on X' which is F/r'-equivariant in 
the sense that for each 7 £ F/F' we are given an isomorphism : ")*{E) — > E (where 
7* is induced by both the action of 7 on X' and on G), and these isomorphisms satisfy 
the compatibility conditions i^^i-ya = ^7172 for all 71,72 £ F/F'. 

We define the Hecke stack J{(G, X) as the algebraic stack which classifies quadruples 

{x,E,E',4>), 
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where x G X, E,E' £ Bunc and (p is an isomorphism 

E\x-{x} — E'\x^{x}- 

Equivalently, it classifies {x,E,E',(f)) where x £ X, E,E' are F/F'-equivariant G- 
bundles on X' (in the above sense), and (j) is an isomorphism 

which is invariant under the F/F'-action on E and E' . 

For every geometric point x of X, the restriction of G to the formal disc around x 
is isomorphic to G, and there is an isomorphism between G\d^ and the constant group 
scheme over with the fiber G, well-defined up to the finite action of F on G. It 
follows that the morphism 'K{X,G) X x Bunc sending the quadruple {x,E,E',(j)) 
to (x, E) is a fibration over X x Bunc with fibers over geometric points isomorphic to 
the affine Grassmannian Gr = G((t))/G[[t]]. 

Furthermore, if E and E' are G-torsors over equipped with an isomorphism over 
the punctured disc D', then inv3:(£^, £") is well-defined as an orbit [jj] of F on X+. 
Since we have a partial order on the set [X+ /F] of orbits of F on X+ , it makes sense 
to write mVx{E, E') < [/i]. Hence, for every orbit [/u] of F acting on X+, we define the 
closed substack "K^^^ of by imposing the inequality 

(2.5) mv,iE,E') <[fi]. 

It is again a fibration over X x Bunc with fibers isomorphic to the union Grj^] of 
Gr^ C Gr with ^ in the F-orbit [fi]. 

The following description of 'K(X,G) will be useful in Section |2.5[ Let be the 
stack which classifies quadruples {x' , E , E' , (j)) where x' G X' , E is a F/F'-equivariant 
G-bundle on X', E' is a G-bundle on D^', and (p is an isomorphism 

E\d; -E'\d;. 

x' x' 

The morphism 

5{ ^ X' X Bunc 

sending (x', E, E' , (p) to (x', E) makes it into an ind-scheme over X' x Bunc whose fibers 
are isomorphic to the affine Grassmannian Gr of G. We also have a natural action of 
F/F' on 'K lifting the action on X' , and 

(2.6) :K{X,G) ^J</{T/T'). 

2.3. Symmetric powers. For every positive integer d, we introduce a sort of dth 
symmetric power of !Kj^] . Let X^ be the dth symmetric power of X which is defined as 
the scheme-theoretic (also known as GIT) quotient of X'^ by the action of the symmetric 
group Sd- Thus, Xd classifies effective divisors of degree d on X. This is a smooth 
algebraic variety (see, e.g., [S], p. 95). 

The algebraic stack '^d,[fi] over k classifies the data 

(2.7) iD,E,E',(t>), 
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where 

r 

(2.8) D = Y,n^[xi] 

i=l 

is an effective divisor on our curve X of degree d (equivalently, a point of Xd)-, E and 
E' are two principal G-bundles on X, and (j) is an isomorphism between them over 
X — supp(L'), such that for each i the inequahty 

(2.9) imi^^{E,E')<[ni^] 
is satisfied. Consider the morphism 

^ X Bunc 

sending the quadruple (|2.7|) to {D,E). Its fiber over a fixed D = '^^riilxi] and E G 
Buug isomorphic to the product 



i=l 

The algebraic stack 'Kd,[fi] is a closed substack of the dth symmetric power of the 
Hecke stack which is defined as the classifying stack of the quadruples (j2.7p as above, 
but with the condition inv^. (i?, £") < [rij^u] removed. To verify that these conditions 
define a closed substack, it is convenient to introduce a chain version of ^[^j . Let j^^j 
be the algebraic stack classifying 

(2.11) {x„E„cj),), 

where 

• X, = (xi, ... ,Xd) G X''-, 

• E, = {Eo,..., Ed) with Ei £ Bunc, 

• 0, = (01, . . . , (/)(i) where (pi is an isomorphism 

Ei-i\x~{x,} - -£^iU-{xJ 
such that 'mVxi{Ei-i, Ei) < [fj]. We have a morphism 



(2.12) 4 : ?{f^] 



sending (x.,^,,(/).) to {D, E, E' , cj)), where D = EiLil^^d; E = Eq, E' = Ed, and 
c;^ is obtained by composing cpi, . . . ,(j)d on their common domain of definition X — 
supp(D). The morphism ()2.12p is proper and ^d,[ii] is its image; its geometric points 
are characterized by the inequality (12. 9|) . 

We also have a chain version Jt*^ of the entire Hecke stack ^K, which classifies the same 
objects as above, but without the condition that mVxiiEi_i, E^) < [/j,]. The morphism 
((21^ extends to 

„d . ard , rrr 
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2.4. Examples. Let us discuss examples of the above construction. Let G be the 
multiphcative group Gm over X. Recah that Gm-pi"iiicipal bundles on X are equivalent 
to line bundles, or invertible Ox-modules. Their classifying stack Pic has a well-known 
structure. The degree of a line bundle induces a bijective map from the set of connected 
components of Pic to Z. The kernel of this map, denoted by Pico, classifying line bundles 
of degree 0, is isomorphic to the quotient of an abelian variety ~ namely, the Jacobian 
Jac of X - by the trivial action of Gm (which is the group of automorphisms of line 
bundles) 

In this case, the reduced part of the affine Grassmannian is the discrete set Z (how- 
ever, the non-reduced structure at each point is highly non-triviajl) . The Hecke stack 
"K classifies quadruples 

{x,L,L\(t)), 

where {x,L,L') G X x Pic x Pic and (p is an isomorphism 

L\x-{x} — L'\x-{x}- 

At the level of A;-points we then have L' = L{fix) for some integer E Hence the 
reduced part of 'K is 

Pic xX xZ 

in this case. For every G Z, which is the set X_|_ in this case, we have the inclusion 
(of the reduced parts, which we denote by the same symbols by abuse of notation) 

Jif, = Pic xX X {fi} c Ji. 

The other projection onto Pic is {L,x,fj.) ^ L' = L{fj,x). 
Since Gm is commutative, can be identified with 

J£a = Picx{X X Z)d 

where {X x Z)^ denotes the set of orbits of the symmetric group Sd on {X X Zf. We 
have two maps {X x Z)^ — )• and {X x Z)d —?■ X^. The dth symmetric power of 
Z consists of elements of the form ((di, /xi), . . . , {dr, fir)) where di, . . . ,dr are positive 
integers satisfying di + ■ ■ ■ + dj- = d and /ii < • • • < /x,. is a strictly increasing sequence 
of integers. The fiber of {X x Z)d over ((di,/xi), . . . , (4,/x,.)) is X('^i) x • • • x X'^'^-) 
projecting onto Xd by the obvious map of adding the divisors. In particular, for the 
element (d, /x) G Z^, we have the component 

'^d,^, = Pic y-Xd- 

Our second example is a non-split one-dimensional torus H over X. Such a torus 
is given by a /X2-torsor (equivalently, etale double cover) vr : X' — > X. Let us denote 
by r the non-trivial involution on X' . The group P' C P has index 2 in this case 
and r/r' = {l,r}. We assume that X' is geometrically connected so that H remains 
non-split over X An i/-torsor on X is a line bundle L over X' whose norm down 
to X is equipped with a trivialization /3 : Nx>/x{L) — > Ox- 

^Indeed, for a general fc-algebra R the quotient R{{t))/R[[t]] is isomorphic to the product of Z and 
the set of polynomials of the form 1 + r-it^^ + r_2i^^ + • • ., where the Vi are nilpotent elements of R. 

^However, this is not true at the level of i?.-points, where _R is a ring with nilpotents, as explained 
in the previous footnote. 
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The group of connected component of Bun// is 'L/2'L. The neutral component of 
Bun// is a quotient of an abelian variety - namely, the Prym variety of X' - by the 
trivial action of Z/2Z which is the group of automorphism of any pair (L, /3) as above. 

The set X_|_ = Z is acted on by the involution t(p) = — /i. The set of orbits [X+/r] 
may therefore be indexed by non- negative integers m G Z>o. The moduli stack "K 
(again, the reduced part) decomposes into connected components 

(2.13) 'K= [_\ 

meZ>o 

with "Kq = Bun// xX and "Km = Bun// xX' for m > 1. The second projection onto 
Bun// may be described as follows: for m = we map {L,x) i— >■ L, and for m > 1 we 
map (L, x') I—)- L{m{x' — t{x'))). 

For every d > 1 we have the following description of Jld' 

(2.14) :Kd = Bun// x (X U (X' x Zyo))d- 

The connected components of (X U (X' x Z>o))(i are the fibers over elements of (Z>o)(i, 
which are r-tuples {{do,mo), {di,mi), . . . , {dr,mr)) with = uiq < mi < ••• < rrir, 
where do > and di, . . . ,dr > are integers such that do + di + ■ ■ ■ + dr = d. The 
corresponding component of Ji^ is 

Bun// xXd,, xX'd^ X ■■■X X'd^. 

2.5. The perverse sheaf. Let p be a finite-dimensional representation of ^G. For 
simplicity, we will restrict ourselves to irreducible representations. If G is split, for 
example, then such p must be of the form p = ® Pl-, where p^ is the irreducible 
representation of G with highest weight p, and p£ is the representation of F attached 
to an irreducible local system £ on X. 

For any semi-simple finite-dimensional representation p of ^G, we will define perverse 
sheaves %p on "K and "Kd^p on IK^. They may be constructed more conceptually using 
an equivalence of categories suggested in the Appendix to [FGV2j . but for our purposes 
the explicit construction given below will suffice. 

First, we assume that G is split and p is an irreducible representation of ^G = G x F 
which is trivial on F. Recall that we have defined a closed substack of "K such that 
the projection X x Bunc is a locally trivial fibration with fibers isomorphic to 

the Schubert variety Gr^. 

Using the geometric Satake correspondence [MVj, we associate to each irreducible 
representation p = p^ oi ^G of highest weight p € X+ the sheaf 

%p = %^ = IC(:K^J[-dim(X x BunG)](- dim(X x BunG')/2). 

With the above cohomological shift and Tate twist, locally over X x Bunc, "Kp is 
isomorphic to 

Q^KIC(&^), 

where IC(Gr^) denotes the intersection cohomology sheaf of Gr^ and is the constant 
sheaf on X X Bunc. Hence "Xp is perverse and pure of weight along the fibers of the 
projection to X x Bunc. By abuse of notation, we will use the same symbol for i*3Cp, 
where i : IK^ — )• so that %p will also be viewed as a sheaf on IK. 
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We define the Hecke functor Hp = as the integral transform corresponding to the 
kernel %p (see [BP] ): 

For X G \X\, let "Kx be the fiber of ^ over x, and Px,p'x ■ ~^ Bun^ the corresponding 
morphisms. Denote by JCp^x the restriction of %p to "Kx- Define the functor Mp^x by 
the formula 

Mp,xm=PxM*i3^)^^P,x). 

The function corresponding to the sheaf JCp^x is the kernel Kp^x of the Hecke operator 
tlp,x and so the functor Mp^x on the derived category D(BunG') of ^-adic sheaves on 
Bunc is a geometric analogue of the Hecke operator Hp a;. 

Next, suppose that G is split and p is a more general irreducible representation of 
= G X F of the form p = Pp ® pc^ where p^ is the irreducible representation of G 
of highest weight p € X_|_ and /)£ is the representation of F attached to an irreducible 
local system L on X. Now we put 

(2.15) Xp = Xp^s*{L), 

where s is the obvious projection onto X. By abuse of notation, we will use the same 
symbol for i^%p where i : "Kp — >■ !K, so that Xp could also be viewed as a sheaf on "K 
(again, it is perverse on the entire only up to a cohomological shift, but perverse 
along the fibers of the projection to X x Bunc). 

Finally, consider the non-split case, following the notation of Section 12.21 We have 
an exact sequence 

l^GxF'^GxF^ F/F' ^ 1. 

Let p be an irreducible representation of G x F. Suppose that its restriction to G x F' 
is semi-simple, so that 

(2.16) = 0(p^®p£)®"M,. 

over a finite collection of pairs {p,L), where p G X+ and L is an irreducible local 
system on X' . Here pp is an irreducible representation of G of highest weight, pc is the 
representation of F' attached to L , and the integer np^£ is the multiplicity. 

We attach to this p a sheaf Xp on IK(X, G) as follows. First, we construct, in the 
same way as above, the sheaf X'p on the stack "K introduced in Section 12.21 using the 
fact that this stack is a locally trivial fibration over X' x Bun^ with fibers isomorphic 
to the affine Grassmannian Gr of G. Since p is a representation of G x F (and not just 
G X F'), we obtain that X'p is a F/F'-equivariant sheaf on Jf. Hence it descends to a 

sheaf on the quotient of "K by F/F', which is nothing but Jf(X, G), by formula (12. 6p . 
This gives us the desired sheaf Xp on !K(X, G). 

We note that this construction yields a generalization of the geometric Satake cor- 
respondence to the quasi-split case. 
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2.6. Symmetric power. Next, we will construct the symmetric power %d,ii of JC^ for 
any positive integer d. 

First, we recall the construction of the symmetric power of a local system L on X. 
From our point of view, this is a particular case of the general construction correspond- 
ing to the trivial G (or general G but ^ = 0) . 

Let £ be a rank n local system on a curve X over a field k (which, as before, is 
either C or F^); that is, a rank n locally constant sheaf (£-adic, if /c = Fg) on X. Recall 
from Section [2.31 that Xd = X'^/Sd is the dth symmetric power of X. This is a smooth 
algebraic variety defined over k, whose fc-points are effective divisors on X of degree d. 

We define a sheaf on Xd, denoted by Ld and called the dth symmetric power of L, 
as follows: 

(2.17) £.= (7r,^(0)^', 

where tt'^ : X'^ Xd is the natural projection. The stalks of Ld c^re easy to describe: 
they are tensor products of symmetric powers of the stalks of L. The stalk Ld^D at a 
divisor D = ni[xi\ is 

'Cd,D = (g)5"^('C.J, 

i 

where S'^^{Lxi) is the nj-th symmetric power of the vector space Lx^- In particular, the 
dimensions of the stalks are not the same, unless n = 1. In the case when n = 1 the 
sheaf Ld is in fact a rank 1 local system on Xd- For all n, Ld is actually a perverse sheaf 
on Xd (up to cohomological shift), which is irreducible if and only if L is irreducible. 

Let denote by the open subscheme of X'^ defined by the equation Xi ^ xj for 
all indices i ^ j. The action of the symmetric group Sd on X'^ preserves X'^'° and the 
quotient of X'^'° by Sd is an open subscheme Xd^o of Xd which classifies multiplicity 
free effective divisors of degree d on X. 

Lemma 2.1. Ld[d] is a perverse sheaf on Xd which is the intermediate extension from 
a local system on Xd^o- 

Proof. The natural projection vr"' : X'^ — )■ Xd is a finite fiat map which is etale over the 
open subscheme Xd^o of Xd- As a finite map, tt'^ is in particular small in the sense of 
Goresky and MacPherson. If p : y — )• Z is a small map and 3" a local system on Y, 
then p*(3")[dimy] is a perverse sheaf on Z isomorphic to the intermediate extension 
of its restriction to any open dense subset of Z. Hence we obtain that this is true for 
the sheaf 7r^(£^"') on Xd- Taking the S'^-invariants, we obtain the statement of the 
lemma. □ 

This lemma suggests the following general construction of 3"d,p- We note that a 
similar construction was proposed by Laumon in |Lau2j in a slightly different context 
(for GLn). Recall that we have a map 
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defined in (I2.12|) . We have in fact a commutative diagram 
(2.18) Mg,! JIIk,,,^, 



which is Cartesian over the open subscheme X^^o- Recall that over the same open 
subscheme, the chain version defined in Section 12.31 is canonically isomorphic to 
the dth Cartesian power of the map 

?{[^] Bunc . 

By restriction, {%p)^'^ defines a perverse sheaf on 

which is equipped with an action of Sd- By finite etale Galois descent, we get a perverse 
sheaf over 

Now, we can apply the functor of intermediate extension to obtain a perverse sheaf 
(2.19) Xd,pOn 

In the above discussion, we have neglected a shift, since defined as in (j2.15p is only 
a perverse sheaf after a shift. Recall that the effect of this shift makes the restrictions 
of %p to the fibers of the map IK[^] — )• X x Bung- perverse sheaves. With the above 
definition, the restrictions of 'yid,p to the fibers of the maps 

over the open subset x Bunc are also perverse sheaves. We will see that this is also 
true for all other fibers. 

The above definition of %d,p is probably the quickest one, but it is impractical for 
many purposes as the intermediate extension is not very explicit. The following lemma, 
essentially proved by Mirkovic and Vilonen in [MVj . uses the whole diagram (12.180 to 
make the definition of 3C(i,p more explicit. 

Lemma 2.2. The morphism vrJ^ of the diagram ()2.18p is small in the stratified sense 
of [MVj . Moreover, for all x, € X mapping to D £ Xd, let :K^. be the fiber of Jf^ 
over X, and ^ the fiber ofK^ over D. Then the restriction o/vr^ to the fibers of 

■^x. ~^ •^d,D 

is semi-small in the stratified sense. 

Assume now till the end of this subsection that G is split, = GxF and p = p^^pc- 
In this case the construction of 'X.d,p can be made more explicit along the lines of 
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Springer's construction of representations of the symmetric group. Consider the closed 
substack IKJ^ of 'K'^ defined in Section 12.31 and the perverse sheaf 

(2.20) ic(:K^)®pr;;,,(£™) 

on 'K'jj^. Next, consider the push- forward 

(2.21) (4),(ic(:K;:)®pr;,,(£™)) 

to '^d,fi which is contained in !K^. It follows from the first assertion of Lemma 12.21 that 
(j2.2ip is a perverse sheaf and is the intermediate extension of its restriction to any 
non-trivial open subset; in particular, to J^d a ^ X'^'°. This defines an action of Sd on 

()2.2ip . By taking the invariant part of this action and making cohomological shift by 
— dim(X'^ X Buug), we obtain another definition of OCd^p- 

(2.22) Xd,p = ((4),(IC(:K^) 0pr;,,£^'^))^' [-((i+dimBunG)](-(d+dimBunG)/2). 

The first assertion of Lemma 12.21 guarantees the equivalence of the two definitions. 

The shift by - dim(X'^ x Bun^) in ([2:20]) makes our sheaf perverse along the fibers 
of the map IKJ^ X'^ x Bun^. According to the second assertion of Lemma |2.2[ the 
restriction of %d,p to the fibers of the map !Kd^p — >■ Xd x Bunc is also perverse. 

Now it is possible to describe the fibers of %d,p using the geometric Satake equivalence 
[MVj . Fix an effective divisor of degree d 

r r 

D = '^ni[xi\ e Xd, Xiy^Xj, y^^ni = d. 

i=l i=l 

Fix a principal G-bundle E G Bunc and choose trivializations of E on the formal 
discs -Da;.. Then the fiber of "K^ over {D,E) is the product of copies of the affine 
Grassmannian attached to the points Xi, 



i=l 



(2.23) :K,{D,E) = 1[Gt,^. 
The fiber of 'Kd^p over {D, E) is 

(2.24) 'Kd,p{D,E) = \{Gi,^^. 



riip- 



The restriction 'Xd,p to this fiber is the external tensor product of spherical perverse 
sheaves 



(2.25) (^pr*S">(IC(Gr,,,^)0£,J, 

i=l 

where S""' is the nj-th symmetric power based on the symmetrical monoidal structure 
of the category of spherical perverse sheaves on the affine Grassmannian. In other 
words, 5'"''(IC(Gr2,.^^) is the perverse sheaf on the affine Grassmannian which corre- 
sponds to the njth symmetric power of p under the geometric Satake equivalence [MV\ 
(prj denotes the projection of (|2.24p onto the ith factor). This formula follows from 
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the second assertion of Lemma 12.21 and from the fact that the definition of Drinfeld's 
commutativity constraint used in [MV] was also based on this lemma. 

Let Kd^p be the function on BunGr(fc) x Jiundk) corresponding to the sheaf 'Xd,p- 
Then it is given by the formula 

(2.26) Kd,p= ^ JJ-f^Sym-i(p),a;,, 

where Kp^x is the kernel of the Hecke operator tlp^x- 

2.7. Example. Consider the case of G = X x SL2. The dual group is = G x F 
where G = PGL2. Here G is defined over C if fc = C and over if is a finite field. 
We fix an irreducible three-dimensional representation 

P = Pp.'Si PE 

of as follows. Let X' ^ X he an etale double cover of X which is geometrically 
connected. Choose a geometric point x' of X' over the given point 2; € X. This gives 
rise to a non-trivial character of order two of T. The first component pp is the 
irreducible three-dimensional representation of PGL2. Let V be the tautological two- 
dimensional representation of GL2. Then we have the following formula for the lifting 
of p^ to a representation of PGL2: 

(2.27) p^ = S^V ^ det{Vy^ . 
If we chose the splitting of SL2 with the diagonal torus 

r = {diag(t,t-i)} 

and the subgroup of upper triangular matrices as Borel subgroup, then p is simply the 
cocharacter of T given by t i-)- diag(t,t~^). 
For every d > 0, JC^^^ classifies the data 

where D G X^ is an effective divisor of degree d, E, E' are rank two vector bundles 
on X with trivialized determinant and (p is an isomorphism of the restrictions of the 
vector bundles E and E' to X — supp(Z)) such that det((/)) = 1, and cp can be extended 
as a morphism of vector bundles E — ?> E'{D). 

The representation pp corresponds to the intersection cohomology complex with the 
appropriate shift 

Xd,p, lC{Jid,p)[-{d + dim(BunG))](-((i + dim(BunG))/2). 

The sheaf 'JCd,p is 

= IC(:Krf,^) 0pr3,^(£d)[-(d + dim(BunG))](-(d + dim(BunG))/2), 

where prjf^ is the projection on Xd and S-d is the dth symmetric power of £, which is 
a rank one local system on Xd of order two corresponding to />£. 

Next, we consider a closely related example for a one-dimensional twisted torus H 
over X as in Section 12.41 The dual group is Gm x F, where F acts through a 
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quotient nu : T — )• Z/2Z, and Z/2Z acts by the formula r(t) = t ^ for the non-trivial 
element r G Z/2Z and t S Gm- Now, we have a homomorphism G 

(2.28) G„ X r ^ PGL2 X r 

mapping Gm to the diagonal torus of PGL2 and an element a € T to the element 

GPGL2 xr 

where wq is the permutation matrix and n//(cj) denotes the image of a in Z/2Z = {0, 1}. 

Consider the representation p = ® oi ^G, where p^ is the irreducible three- 
dimensional representation of PGL2 and /Og is the one-dimensional representation of F 
of order two as above. If we restrict p to ^H, it decomposes into a direct sum 

PH = p\lh = Po® Pi, 

where pi is a two-dimensional representation of and po is the one-dimensional 
representation which is trivial on Gm, and on the factor T is given by the formula 

P0 = PS,® PEh ' 

p^^ being given by the character tlh : F — )• Z/2Z. In particular, pQ is the trivial 
representation if £ = 8,h- 

Assume from now on that £ = Then the representation pH of corresponds 
to the direct sum 

where %i = Q^I^Cq is the constant sheaf on Jio = X x Bun// and ^2 = Q^l^i is the 
constant sheaf on Jii = X' x Bun//, in the notation of (j2.13p . For every positive integer 
d, we have 

^d,pH = Q^KBun// xXd, X X^J, 

d()+di=d 

where the sum is over all non-negative integers do and di such that do + di = d. Here 
Bun// xXdg X X'^^ is a connected component of "Kn^d described in (|2.14p . 

2.8. Integral transforms. We have defined a morphism 

"K^ Xii X Bung X Bunc 

and a perverse sheaf ^d,p on "K^ attached to any semi-simple finite-dimensional rep- 
resentation p of ^G. Let us denote by pd and the two projections Ji^ Bunc 
mapping the quadruple (|2.7|) to E and E'. We use the sheaf OCd,p to define an integral 
transform functor ^d,p on the derived category L'(BunG) of £-adic sheaves on Bunc by 
the formula 

(2.29) Kd,p(J) = Pdl(.p'd*{^) ® ^d,p)- 

Functors of this kind were first introduced in jFGV2j in the case of GLn under the 
name "averaging functors." An example is the functor Av^ of |FGV2j which plays the 
role of a "projector" on the Hecke eigensheaf corresponding to an irreducible rank n 
local system ii^ on X, in the category of D-modules, or ^-adic sheaves, on Bunc. The 
functors K corresponding to these kernels are combinations of the Hecke functors, in 
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which the positions of the points at which these functors are apphed are allowed to 
vary over the dth symmetric power of X. 

We generalize this construction to a larger family of integral transforms on D(BunG') 
as follows. For any scheme over the symmetric power 

S^Xd 

we form the Cartesian product 

^d,S ~ ^ ^ '^d- 
Ad 

Denote by "^^d.p.s the restriction of 3Crfp to "^ds- Ps ^'^d p'g the two projections 
from '^dS Bunc. Then we have an integral transform 

(2.30) ^d,p,sm = Ps\{p's*i^) ® ^d,p,s). 

If 5 is a point of Xd, this is the usual Hecke functor. If S" = Ud, where U is an open 
subset of X, we have an averaging operator over U that will be useful in the study of 
the ramified case. 

2.9. Vector space. Let us form the Cartesian square 

Md :Kd 

(2.31) PA p 

Bunc — — — 7- Bunc x Bunc 
where A is the diagonal morphism. Thus, is the fiber product of Xd x Bunc and 
"Kd with respect to the two morphisms to Xd x Bun^ x Bunc- 

Let % = '^d,p,s be a sheaf of the type introduced in the previous section and IK the 
corresponding integral transform functor on the derived category D(BunG) of sheaves 
on Buug". Let 

X = p^{%) 

(note that p is proper over the support of X). This is a sheaf on Bunc x Bunc which 
is the kernel of the functor K. 

Note that the sheaf A!(Q^) on Bunc x Bun^ is the kernel of the identity functor Id 
on the category D{BunG)- Hence the vector space 

(2.32) RHom(A!(Q^),5C), 

may be viewed as a concrete realization of the "categorical trace" 

RHom(Id, K) 

of the functor K. 

By adjunction and base change, the space (I2.32P is isomorphic to 

(2.33) H'{BnnG,A-p,{X)) = i7'(BunG, PA*A-,(3<;)) = H' {Md, A^^iX)). 

Let D be the Verdier duality on !K^,^. It follows from the construction and the fact 
that D(IC(:K^)) ~ IC(:K^) that 

(2.34) B{Xd,p^^pJ ^ Xd,p^^pi [2{d + dimBunG)](d + dimBunc). 
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Therefore, up to a shift and Tate twist (and replacing L by £*), the last space in (I2.33P 
is isomorphic to 

(2.35) H'iMdM^ki^))) ~ H:{Ma,A*^{X))*, 

where H'{Z, 3") is understood as fi{3'), where / : Z — t- pt. Here we use the results of Y. 
Laszlo and M. Olsson |L0| on the six operations on £-adic sheaves on algebraic stacks 
and formula (j2.34p . 

Using the Lefschetz formula for algebraic stacks developed by K. Behrend [Be] , we 
find that, formally, the trace of the (arithmetic) Frobenius on 

(2.36) H'iBunG,A-i%)) = /^•(Md, Aj^(X)) 

is equal (up to a power of q) to the sum ()2.3p , which is the right hand side of the trace 
formula (jl.ip . 

For this reason, we will view the vector space (j2.36p as a geometric incarnation of 
the right hand side of the trace formula (II. ip . 

There are some technical issues that have to be dealt with in order to make this 
precise, because is not an algebraic stack of finite type. In general, in order to 
make the trace of the Frobenius on (j2.36p finite, it is necessary to introduce some 
stability condition on the objects that "Md classifies (see Section l4.ip . Imposing this 
stability condition should be viewed as the geometric counterpart of Arthur's truncation 
process of the trace formula (see [CLj , where this is explained in the case of the trace 
formula for Lie algebras), but we will not discuss this issue in the present paper. Here 
we will consider the moduli stack without any stability conditions. 

In Section m we will give a description of the stack M^^ that is reminiscent and closely 
related to the Hitchin moduli stack of Higgs bundles on the curve X [HI] . We will also 
conjecture that Al^(Xd^p) is a pure perverse sheaf on M^. 

If X is a curve over C, then the vector space ()2.36p still makes sense if we consider 
3C as an object of either the derived category of constructible sheaves or of 2)-modules 
on Buug- 

3. Eigenvalues of Kd,p on Hecke eigensheaves 
In this section we apply the functor IK^^p to Hecke eigensheaves on Bunc- 

3.1. Cohomology of symmetric powers. Recall the definition of the symmetric 
power of a local system L on X given in Section 12.61 We generalize this definition 
slightly, by allowing £ to be a complex of local systems on X\!\ The resulting object Ld 
will then be a complex of sheaves on X^- 

Let us now compute the cohomology of X^ with coefficients in L^- By Kiinneth 
formula, we have 

H-{Xd,Ld) = [H-iX^,L^'')y' = (^H-{X,Lr''y\ 



The reason for this will become clear in Section [3.41 
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where the action of Sd on the cohomology is as follows: it acts by the ordinary transpo- 
sitions on the even cohomology and by signed transpositions on the odd cohomology. 
Thus, we find that 

(3.1) H'iXd,Ld)= S''\H\X,L))^A^^iH\X,L))^S^^{H\X,L)). 

do+di+d2=d 

This is true for any complex of local systems. The cohomological grading is computed 
according to the rule that do does not contribute to cohomological degree, di contributes 
di, and ^2 contributes 2^2- In addition, we have to take into account the cohomological 
grading on L. 

3.2. Connection to L-functions. Suppose that k = ¥q. Let us form a generating 
function 

(3.2) ^Tr(Fr,i/'(Xd,£d))t^ 

where Tr(Fr , H' {Xd, £jd)) is the trace of the Frobenius on H*{Xci,£jd), thus the alter- 
nating sum of traces on the groups W {Xd,Ld)- 
There are two ways to compute it. 

I. By the Lefschetz formula, 

TT{Fi,H'{Xd,J:d))= Yl Tr(Frz5,£z5), 

where is the stalk of Ld at D = J2i ^d^i], where Y2i cleg(a;j) = d. Since 

i 

we find that the generating function p.2p is equal to 

(3.3) H det{l-t''^^^^^Fv,,L,)-\ 

x€\X\ 

where \X\ is the set of closed points of X. 

If we set t = in formula l\3.3\i . we obtain the L- function L(L,s) of the Galois 
representation associated to L (and defining representation def of GLn). Thus we find 
that TriFr, H'{Xd,Ld)) is the coefficient of q-'^'' in L{L,s). 

II. Formula (|3.ip gives us the following expression for the generating function ()3.2p : 

detjl -t Ft, H^{X,L)) 

det(l -tFr,FO(X,£)) det(l - t Fr, ^^(X, £)) ' 

If we substitute t = q^", we obtain the Grothendieck-Lefschetz formula for the L- 
function -L(£, s). 
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3.3. Hecke eigensheaves. Recall that X is a geometrically connected smooth proper 
curve over a finite field k = ¥q, and G a reductive group scheme over X which is a 
quasi-split form of a constant group G. The Langlands dual group of G, defined over 
Qi, is ^G = G XI r where T is the (etale) fundamental group F = 7Ti{X, x) that acts on 
G by the data defining the quasi-split form G. 

Let Wp be the Weil group of the function field F of the curve X. An Arthur 
parameter is an equivalence class of homomorphisms 

(3.4) (j:SL2xWf^ ^G 

that induces the canonical map Wp Gal(F/F) T. According to the conjectures of 
Langlands and Arthur, equivalence classes of irreducible automorphic representations 
of G(A) may be parametrized by Arthur parameters. Such a representation vr is called 
tempered (or Ramanujan) if 0"|sl2 is trivial. Otherwise, it is called non-tempered (or 
non-Ramanujan) . 

If an automorphic representation 

vr = (g) ' vr^ 

xG\X\ 

of the adelic group G(A) has the Arthur parameter a, then for all closed points x G |X| 
where tt^ is unramified the restriction a\wp is also unramified and the Satake parameter 
of is equal to the conjugacy class 

Now we discuss a geometric analogue of this - in the unramified case, in order to 
simplify the discussion. The geometric analogue of the notion of automorphic repre- 
sentations (or, more precisely, the corresponding spherical automorphic function) is the 
notion of Hecke eigensheaf which we now recall. 

Let yO be a continuous representation of on a finite-dimensional vector space V 
and let a be an Arthur parameter. Then we obtain a representation pofj of SL2 x Wp 
on V. The standard torus of SL2 C SL2 x Wp defines a Z-grading on po a: 

poa = 0(/)Of7)i, 

where each p o (Ji is a continuous representation of Wp. Assume that each of them is 
unramified. Then it gives rise to an ^-adic local system Li^p^^-^. on X. Now we define 
-^poCT to be a complex of local systems on X with the trivial differential 

■^poo- = -^(pocr); [~^]- 

Recall the Hecke functors Hp and Hp^a; introduced in Section 12.51 (geometric ana- 
logues of the Hecke operators corresponding to p). The following definition is a slight 
generalization (to the case when 0"|sl2 is non-trivial) of the definition given in [BP] . 
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Definition 3.1. A sheaf 9" on Bunc is called a Hecke eigensheaf with the eigenvalue a 
()3.4p if for any representation p of we have an isomorphism 

(3.5) IHp( J) ~ Lpoa ^ 3", 

and these isomorphisms are compatible for different p with respect to the structures of 
tensor categories. 

It follows from the above identity that for every x € X, we have an isomorphism 

(3.6) EIp,,(J) ~£po<x,x®:F, 

where £poo-,x is the stalk of p o o" at x. Formula p.6p follows from ([33]) by base change 
for the cohomology with compact support. 

In X is a smooth projective connected curve over C, then the above definition also 
makes sense if we take as cr a homomorphism SL2 x 7ri{X,x) — )• and as £(poCT)i the 
corresponding local system on X. 

Now we compute the "eigenvalues" of the integral transform IC^^p on 3"(j. The fol- 
lowing result is Lemma 2.6 of [FLN]. 

Lemma 3.2. If J = "J^ is a Hecke eigensheaf with eigenvalue a, then for every repre- 
sentation p of and every positive integer d we have 

(3.7) IKd,p(:J^) = H*{Xd, £po.,d) 5-<x. 

Proof. We will restrict ourselves to the split case. The general case is obtained from 
the split case by the descent method. From now on, the group scheme G is of the form 
G X X, ^G is G X r, the representation p is assumed to be of the form p = p^ ^ pz 
where is a irreducible representation of G of highest weight p and p£ is the continuous 
representation of F attached to an irreducible local system L over X. 

Consider the chain version of the Hecke stack from Section 12.31 with truncation pa- 
rameter p, 

Jif^^X"^ X Bunc X Bunc, 
with the (shifted) perverse sheaf 

3c^ = ic(:K;:)®pr;,,(£™). 

As in Section [2T8I let us denote by pd, p'^ the two projections to Bunc and = ^y^xd xp^, 
where prj^d is the projection to X'^. Consider the integral transform 

i:»(BunG) ^ DiX"^ X Bunc) 

given by 

H^:^^9d!(Pd*^?®3^p)- 
By using repeatedly ()3.5p . we obtain 

By pushing forward along X'^, we get 

(3.8) prBunG,!EIp(:?<x) = ^•(X,£po.)®'' S'a- 
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The symmetric group Sd acts on the right hand side in the obvious way and the invariant 
part is 

H'{Xd, ^pou,d) S'cr- 

On the other hand, recall the morphism 

. nrd , rrr 

By definition, the perverse sheaf 'X.d,p is the invariant part of Tr^,%p under the action 
of Sd- Hence the the S'^-invariant part of the left hand side of (|c{.8p is Kd^p{3'a), and 
we obtain the desired formula ()3.7p . □ 

Corollary 3.3. For all d > {2g — 2) dim p, we have ^d,pi^a) = 0, unless poa or (poa)* 
has non-zero invariants under the action ofWp (or iTilXjx), if X is defined over C). 

Proof. Indeed, ii po a and (poa)* have zero spaces of invariants, then 

H°{X, poa) = H'^{X, poa) = 

and dim {X, poa) = {2g — 2) dim p (since it is then equal to the Euler characteristic 
of the constant local system of rank dimp on X). Hence we obtain that 

H'{Xd,Lpo^^d)-^''{H\X,poa)), 

and the statement of the corollary follows. □ 

We can translate the above lemma and corollary to the level of functions: the eigen- 
value of the operator p (see Section I2.8P on an unramified Hecke eigenfunction /„- is 
equal to the ^"'^^-coefficient of the automorphic L- function L{p,a,s). This can indeed 
be shown directly at the level of functions as follows (see also the proof of Lemma 2.6 
in [FLN] V 

When Kd^p acts on 9", for a fixed effective divisor D = X]jni[xi] of degree d we 
apply modifications at the closed points Xj. These modifications are described by the 
Schubert varieties Gr^^^ with the "kernel" /^"'(^(Gr^)), which is the perverse sheaf 
on Gr„.^ corresponding to the njth symmetric power of the representation p. 

When we apply the Hecke functor corresponding to 5"' (IC(Grp)) at the point Xi to 
our sheaf 3", we simply tensor it by S'^^{p o a)xi-, because 3" is a Hecke eigensheaf with 
eigenvalue a. So the net result is that for each fixed D we tensor 3" with 

®S-^{poa),^ 

i 

(note that in general this is a complex of vector spaces). But this is precisely the 
stalk of (p o a)d at D (see formula (j2.17p ). In other words, because we are applying 
Krf^p to a Hecke eigensheaf 3", we are effectively replacing each Hecke operator by the 
corresponding "eigenvalue" , that is, S"' (p o (t)^^ ■ 

Then we need to integrate over all possible D. This simply means taking the coho- 
mology of Xd with coefficients in (p o a)d- The result is 

Kd,p{3^)^H'{Xd,{po(^)d)^3', 

as in Lemma [3.21 Now, each step of this calculation makes sense at the level of functions. 
Hence we obtain that the eigenvalue of the operator K.d,p on is equal to the trace 
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of the Frobenius on H'{Xd, {p o a)d), which is the g~ -coefficient of the automorphic 
L- function L{p,a,s), as we have seen in Section [3.2[ 

li poa and (poa)* have zero spaces of invariants, then L(a, def, s) is a polynomial of 
degree {2g — 2) dim p in and so K.d^p{fa) = in this case for all d > {2g — 2) dim p. 
This is function-theoretic version of Corollary 13.31 

3.4. Example: constant sheaf. Assume that G is split, = G x F, /? = (g) 

po, where p^ is the irreducible representation of highest weight p, po is the trivial 
representation of F. In this case JCp = %p is the intersection cohomology complex of 
"Kp shifted by — dim(X x Bunc). 

Let 3'q = Qi \ Bunc, the constant sheaf on Bunc. This is the geometric analogue of 
the trivial representation of G{A). Let us apply the Hecke operator M^^^^ to 3"o- For 
any G-principal bundle E, the fiber of (iJix is isomorphic to Gr^;, once we have 

chosen a trivialization of E on the formal disc Dx- Thus the fiber of Mp^xi^'o) at E is 
isomorphic to 

Mp^x{3')E = H'{Gip,lC{Gip)). 

This isomorphism is actually canonical in the sense that it does not depend on the 
choice of the trivialization of E on Dx, so that the above isomorphism can be put in a 
family with respect to the parameter E. Hence we obtain that 

^pA^o) - H'(G^p,lC{G^p)) Jo- 

By the geometric Satake correspondence |MV| . 

H'{-GTp,lC{G-T^))c,pf[, 

a complex of vector spaces, which is isomorphic to the representation pp with the 
cohomological grading corresponding to the principal grading on pp. In other words, 
this is {p o (7o)x, where ctq : Wp x SL2 — )• ^G is trivial on Wp and is the principal 
embedding on SL2. We conclude that the constant sheaf on Bun^ is a Hecke eigensheaf 
with the eigenvalue ctq. This is in agreement with the fact that (Tq is the Arthur 
parameter of the trivial automorphic representation of G{A). 

For example, if pp is the defining representation of GLn, then the corresponding 
Schubert variety is P"~^, and we obtain its cohomology shifted by (n — l)/2, because 
the intersection cohomology sheaf IC(Gr^) is the constant sheaf placed in cohomological 
degree — (n — 1), that is 

H'{G^p, IC(G?^)) = 0{«~3)/2 ... 0-in~i)/2^ 

where 6*1/2 ^ q^[_i](_i/2). This agrees with the fact that the principal grading takes 
values (n — l)/2, . . . , — (n — l)/2 on the defining representation of GLn, and each of the 
corresponding homogeneous components is one-dimensional. 
Next, we find that 

^d,p,{Qi) - H'{Xd, {pp o ao)d) ^ Qe, 



26 



EDWARD FRENKEL AND NGO BAG CHAU 



where p^^octq is the complex described in the previous section. At the level of functions, 
we are multiplying the constant function by 

(3.9) Tr(Fr,//-(X,,(p^oao),))= J] C(s-if 

ieP{pM) 

where P{pn) is the set of possible values of the principal grading on and p^^i is the 
corresponding subspace of p^. 

For example, if is the defining representation of GL„, then formula ()3.9p reads 

n-l 

(3.10) J]^(s + A:-(n-l)/2). 

3.5. Decomposition of the trace. Now we wish to use Corollarv 13.31 to decompose 
the trace of ^d,p for d > (2^ — 2) dim p as a sum over subgroups of ^G. 

Let a be an Arthur parameter. We attach to it two subgroups of ^G: ^G = ^G^ is 
the centralizer of the image of SL2 in ^G under a, and = ^H^, is the Zariski closure 
of the image of Wp in "^Go- under a. The representation po a has non-zero invariants 
if and only if the restriction of p to ^Hfy has non-zero invariants. Thus, assuming that 
Arthur's conjectures are true, we obtain that the trace of ^d,p decomposes as a double 
sum: first, over different homomorphisms (f) : SL2 — > ^G, and second, for a given (j), 
over the subgroups of the centralizer ^G^ of having non-zero invariants in p: 

(3.11) TvKd,p = J2 E ^^^H- 

The summands $triv ^/f correspond to the Ramanujan representations. Denote their 
sum by (TrK^^p)/?. 

Note that if (f) is non-trivial, then the rank of ^G^ is less than that of ^G. As 
explained in |FLN| . we would like to use induction on the rank of ^G to isolate the 
Ramanujan part (TrKrf p)i^ in TrK^ p. In other words, we need to isolate and remove 
the contribution of the non-Ramanujan representations. In |FLN] it was shown how to 
isolate the contribution of the trivial representation. 

We then wish to decompose (Tr K^^ p) r over ^H, 

(3.12) (TrKrf,,)^ = (Tr<p,)ii. 

Here the sum should be over all possible '^H C ^G such that '^H has non-zero invariant 
vectors in p o cr, and K^^^ is the operator corresponding to pn = p\^h fo^' the group 
H[A). Proving formula (I3.12|) is the main step in the strategy to prove functoriahty 
outlined in |FLN] . More specifically, we would like to use the orbital side of the trace 
formula ([□]) to estabhsh (IXT2D . 

Actually, comparisons of trace formulas should always be understood as comparisons 
of their stabilized versions. Therefore the traces in (13.121) should be replaced by the 
corresponding stable traces (see [FLN] for more details). Since the fundamental lemma 
has been proved [N3j , the connection between the actual trace formula and the stabilized 
trace formula is now well-understood. 
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The spectral side of (I3.12p (again, assuming Arthur's conjectures) is the sum of the 
eigenvalues of ^d,p which are expressed in terms of the coefficients of the L-function of 
the corresponding Arthur parameter a : Wp — ^H, for different '^H. A precise formula 
for these eigenvalues is complicated in general, but we can compute its asymptotics as 
d — oo. If we divide K^ p by q"^ (geometrically, this makes sense because of the 
dimension of X^i), then the asymptotics will be very simple: 

(3.13) ,-(TVK..,),~ E ^A'Tu)-'!')' 

where N(j is the multiplicity of automorphic representations in the corresponding L- 
packet. 

Indeed, the highest power of q comes from the highest cohomology, which in this 
case is 

H^\Xd, {p o a)d) = Sym\H\X, p o a)) 

{do = 0, di = and d2 = din the notation of formula ()3.ip ). We have dim H^{X, poa) = 
nia (p) , the multiplicity of the trivial representation in poa (we are assuming here that 
this trivial representation splits off as a direct summand in poa), and 

(3.14) dimSym'{H\X,poa)) = ('^^^^j^l^ ' 

Thus, as a function of q'^, the eigenvalues of q^'^K.d,p on the Ramanujan representa- 
tions grow as 0(1) when d — t- oo. For the non- Ramanujan representations correspond- 
ing to non-trivial (j) : SL2 — )• ^G, they grow as a higher power of q"^. For instance, the 
eigenvalue corresponding to the trivial representation of G{A) (for which i;^> is a prin- 
cipal embedding) grows as 0{q'^M), where p is the highest weight of p. In general, 
it grows as 0{q'^), where 2a is the maximal possible highest weight of the image of 
SL2 C under cf) acting on p. Thus, the asymptotics of the non-Ramanujan represen- 
tations dominates that of Ramanujan representations. This is why we wish to isolate 
the Ramanujan part first and then decompose it over '^H. 

The main question we want to answer is the following: how to observe the decom- 
positions (|3.11|) and (j3.12p on the orbital side of the trace formula, using geometry? 

3.6. Example of SL2 and the twisted torus continued. Let us go back to the 
case that G = X x SL2 and the Langlands dual group is ^G = PGL2 x F. We 
consider the representation p = p^^ (E> pe of ^G, where p^ is the three-dimensional 
adjoint representation of PGL2 and £ is a non-trivial rank one local system over X of 
order two which is non trivial. We are now looking at the effect of the operator IK^^p 
on a Hecke eigenfunction /o- attached to an Arthur parameter a : SL2 x F — )• ^G. 

Consider first the case of the Arthur parameter a = aQ whose restriction to the factor 
SL2 is non-trivial, and hence an isomorphism. It then follows that the restriction of 
a to Wp must be trivial. The automorphic representation of SL2(A) attached to this 
parameter is the trivial one-dimensional representation. In this case 3^poa is the graded 
local system 

^poao = (Q£[2] e Of e Q^[-2]) ® £, 
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that is, the direct sum of three copies of £ put in cohomological degrees —2,0 and 2. 
Since H^{X, £) = for i ^ 1, we have 

H-{X, Jpo^o) = H\X, £) ® (Q,[l] e Qe[-l] © Qe[-3]), 

the direct sum of three copies of H^{X,E,) put in degrees —1,1 and 3. In particular 
H'{X,3'poao) vanishes in even degrees. It fohows from an obvious generahzation of 
formula (13. ip that for a large integer d, we have 

H'{Xd,9'd,poao) = 0. 

Hence we obtain that the integral transform ^d,p annihilates the contribution of the 
trivial representation of SL2(A), which is the only non-Ramanujan representation in 
this case. Thus, only Ramanujan representations of SL2(A) contribute to the trace 
formula for large d in this case. 

Now we look at the Arthur's parameters a whose restriction to SL2 factor is trivial. 
The restriction of a to Wp is of the form 

a{a) = (cr+(a), <pia)) € G x F, a G Wp, 

where ip : Wp — ?• T is the canonical homomorphism, for some homomorphism o"+ : 
Wp — )• G. The local system £poo- is of the form 

where £p^oo-+ is the rank three local system given by the representation of Wp given 
by the composition of cj-|_ : Wp — )■ PGL2 and the three-dimensional representation p^J^ 
of PGL2. 

If either H^\X,Lpoa) or H'^{X,Lpoa) is non-zero, then the three-dimensional rep- 
resentation p o a must have either non-zero space of invariants or coinvariants. Since 
'^p^oa is semi-simple because of the purity, p o a has non-zero invariants. Let ^B. be 
the closure of the image of cr+. We want to understand when this subgroup has an 
invariant vector v in the adjoint representation V of PGL2. 

We identify PGL2 with the group SO3 preserving the Killing form on V . There are 
two possibilities for the invariant vector: be isotropic or anisotropic. 

If the invariant vector is v is isotropic, v belongs to the two-dimensional vector 
space (f)"*". Since we have assumed (T-|_ to be semi-simple, there exists another vector 
v\ ^ (f)-*- such that aj^{Wp) preserves the line (vi). Let V2 be a vector generating the 
one-dimensional vector space (v)"*" R (fi)"*". We have a decomposition of V into lines 
preserved by Oj^{Wp) 

y = (v) © (vx) © (^2). 

One can check that aj^{Wp) is contained in the split torus SO2 C SO3. This means 
that cr is a parameter of an Eisenstein series, so that the rank three local system Lp^oa+ 
is of the form 

^p,oa+ = £1 ©£r^ ®Ql 

for some rank one local system £1 on X. 

If £p^o(7 = ■^Pfj0(7+ (8'£ contains the trivial local system, the only possibility is £ = £1, 
so that 

'Cp^oa = Q£©Q£©£. 



GEOMETRIZATION OF TRACE FORMULAS 



29 



If the invariant vector v is anisotropic, then v ^ {v) and we have an orthogonal 
decomposition of V 

v = {v)e{v)^. 

In this case, a+{WF) is contained in the subgroup O2 of SO3, 

O2 = (0(1) X O2) nSOs. 

If a+{WF) is contained in SO2, we end up in the previous case. If it is not contained 
in SO2, then induces a surjective homomorphism /Og^ : Wp — > Z/2Z corresponding 
to a certain non-trivial rank one local system £1 of order two. In this case ^Cp^oa^ has 
the form 

where Vi is a rank two local system of determinant detCVi) = £1. If the image of 
(7+ contains only two elements, then we again end up in the same situation as above, 
i.e., Vi = © £1 and a is the Arthur parameter of an Eisenstein series that we have 
encountered above. 

Finally, if the image of cr+ contains more than two elements, then Vi is an irreducible 
rank two local system. Now if 

^p,oa = (£1 ® £) © (Vl £) 

contains the trivial local system, the only possibility is £1 = £, because Vi (81 £ is an 
irreducible rank two local system. This means that a factors through the dual group 
of the twisted one-dimensional torus H = attached to £. 
To summarize, apart from the Eisenstein series, only (cuspidal) automorphic repre- 
sentations with the Arthur parameter factorizing through the dual group of the 
twisted one-dimensional torus contribute to the trace of the operator K^^ p. Actu- 
ally, the parameter of the Eisenstein series which contributes to the trace also factors 
through as follows: 

a : Wf Gm yi Wf PGL2 X r, 

where Wf — > Gm x T is given by a — t- pe.{oi) x (p{a). The contribution of these 
representations must be exactly the same as the trace of the operator ^d,pH ^he 
twisted torus H corresponding to the restriction pn of p from to ^H. 

4. The moduli stack of G-pairs 

As we have seen in Section 12.91 the right hand side of the trace formula (jl.ip is equal 
(up to a power of q) to the trace of the (arithmetic) Frobenius on the vector space 

(4.1) H'{Md,A'^i%d,p)). 

Recall from Section [2^ that the sheaf %d,p is supported on the substack IK^j^j of "K. In 
this section we will mostly consider the split case, so in order to simplify our notation 
we will write 'Jid,p for ^H^dj^j. Let M^ ^ be the fiber product of x Bunc and IK^^ ^ 
with respect to the two morphisms to x Bunc x Bun^. In other words, we replace 
"Kd by ^d,ii in the upper right corner of the diagram (j2.3ip . The sheaf A'>^{%d^p) is 
supported on M^^^ C M^, and hence the vector space (|4.ip is equal to 

(4.2) H'iMd,p,A:^{%d,p))- 
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In this section we show that the stack "M^^fj, has a different interpretation as a moduh 
space of objects that are closely related to Higgs bundles. More precisely, we need 
"group-like" versions of Higgs bundles (we will call them "G-pairs"). The moduli 
spaces of (stable) Higgs bundles has been introduced by Hitchin [HI] (in characteristic 
0) and the corresponding stack (in characteristic p) has been used in |N3j in the proof 
of the fundamental lemma. In addition, there is an analogue ^d,^ of the Hitchin base 
and a morphism hd,f_i '■ 'M.^^fj, ^d^i analogous to the Hitchin map. We would like to 
use this Higgs bundle-like realization of M^^^ and the morphism hd,^ in order to derive 
the decompositions (j3.1ip and (I3.12j) . 

4.1. Definition of the moduli stack. Let us assume that G is split over X and n is 
a fixed dominant coweight. The groupoid Mrf^^(A;) classifies the triples 

iD,E,ip), 

where D = ^ • [xi] G is an effective divisor of degree d, £' is a principal G-bundle 
on a curve X, and ip is a section of the adjoint group bundle 

Ad{E) = ExG 

G 

(with G acting on the right G by the adjoint action) on X— supp(I?), which satisfies 
the local conditions 

(4.3) 'va^xii^) < nm 

at D. Since we have defined IK^^^ as the image of 'K'j^ in IK^, it is not immediately clear 
how to make sense of these local conditions over an arbitrary base (instead of Spec(/c)). 
There is in fact a functorial description of "K^^^ and of M^,^ that we will now explain. 

We will assume that G is semi-simple and simply-connected. The general case is not 
much more difficult. Let wi, . . . ,0;^ denote the fundamental weights of G and 

p^^:G^GUV^^) 

the Weyl modules of highest weight Wj. Using the natural action of G on End(V^.), we 
can attach to any G-principal bundle E on X the vector bundle 

(4.4) End<^,(^) = ^xEnd(K;J. 

G 

The section if of Ad(£') on X— supp(D) induces a section End^jj {^p) of the vector bundle 
End(^^(£^) on X— supp(-D). The local conditions ()4.3p are equivalent to the property 
that for all i, End^j. (99) may be extended to a section 

G End<^,(^) ®0x Ox{{l^,uJi)D). 

Though the ipi determine (p, we will keep if in the notation for convenience. 

Thus, we obtain a provisional functorial description of M^^^ as the stack classifying 
the data 

(4.5) {D,E,ip,ipi) 

with D € Xd, E € Bunc, (/? is a section of Ad{E) on X—supp{D), ipi are sections of 
End^. (i?) ®0x {{1^,1^1)0) over X such that 

^i\x-supp(D) = Endaj,((/9). 
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Sometimes it will be more convenient to package the data as a single object 

cp which has values in the closure of 

r 
i=l 

where g (z G and ti, . . . ,tr € Gm are invertible scalars. This way Vinberg's semi-group 
[Vij makes its appearance naturally in the description of M^^^. We will discuss this in 
more detail elsewhere, and for the time being will stick to the more concrete description 
(j4.5p of the moduli space. 

4.2. Comparison with the Hitchin fibration. It is instructive to note that the 
stack M,^^^ is very similar to the moduli stacks of Higgs bundles (defined originally 
by Hitchin [HI] and considered, in particular, in [ N21 IN3j ). The latter stack Jsf^) also 
depends on the choice of an effective divisor 

D = y^^riilxj] 

i 

on X and classifies pairs {E, (p), where E is again a G-principal bundle on X and (p is 
a section of the adjoint vector bundle 

ad{E) = E XQ 

G 

(here g = Lie(G)) defined on X— supp(D), which is allowed to have a pole of order at 
most Tii at Xi. In other words, 

(t)€ H^{X,ad{E)(g)(Dx{D)). 

This (j) is usually referred to as a Higgs field. 

In both cases, we have a section which is regular almost everywhere, but at some 
(fixed, for now) points of the curve these sections are allowed to have singularities which 
are controlled by a divisor. In the first case we have a section ip of adjoint group bundle 
Ad{E), and the divisor is D ■ fj,, considered as an effective divisor with values in the 
lattice of integral weights of ^G. In the second case we have a section cj) of the adjoint 
Lie algebra bundle ad(i?), and the divisor is just the ordinary effective divisor. 

An important tool in the study of the moduli stack is the Hitchin map |H2] from 
l^o to an affine space 

AD^^H^{X,(Dx{{m, + l)D), 

i 

where the m^'s are the exponents of G. It is obtained by, roughly speaking, picking the 
coefficients of the characteristic polynomial of the Higgs field (j) (this is exactly so in 
the case of GL„; but one constructs an obvious analogue of this morphism for a general 
reductive group G, using invariant polynomials on its Lie algebra). A point a ^ Ae 
then records a stable conjugacy class in q{F), where F is the function field, and the 
number of points in the fiber over a is related to the corresponding orbital integrals in 
the Lie algebra setting (see |N2l IN3] ). 

More precisely. Ad is the space of section of the bundle 

i/W X Ox{DY 
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obtained by twisting t/W = Spec(/c[t] ), equipped with the Gm-action inherited from t, 
by the Gm-torsor (Dx{D)^ on X attached to the hne bundle (Dx{D). Recall that fc[t/VF] 
is a polynomial algebra with homogeneous generators of degrees di + 1, . . . , dj. + 1. 

In our present setting, we will have to replace t/W by T/W. Recall that we are 
under the assumption that G is semi-simple and simply-connected. First, recall the 
isomorphism of algebras 

k[Gf = k[T]^ = k[T/W]. 

It then follows from [Bou] , Th. VI. 3.1 and Ex. 1, that k[G]'^ is a polynomial algebra 
generated by the functions 

where uji, . . . ,ujr are the fundamental weights of G. 

For a fixed divisor D, the analogue of the Hitchin map for Md^^(-D) (the fiber of M^,^ 
over D) is the following map: 

r 

(4.6) MdAD)^^H'>{X,Ox{{f^,uji)D)) 

i=l 

defined by attaching to {D, E, ip, ipi) the collection of traces 

triipi) e H\X,(Dx{{f^,io^)D)). 
By letting D vary in X^, we obtain a fibration 

where Ad,i_i is a vector bundle over Xd with the fiber H^{X, (Dx{{fJ',^i)D)) over 

an effective divisor D € Xd- 

The morphism /i^.^ : M^^^ — )• ^d,^ is very similar to the Hitchin fibration. We 
are now going to outline the geometric properties of the Hitchin fibration that can be 
carried over to our new situation. We will discuss in more detail the case of SL2 and 
postpone the general case for another occasion. 

There exists a Picard stack CP^^^ — )• Ad,fj, that plays the same role as T ^ A con- 
structed in [N2]. In particular, for every a G Ad,f_i, the Picard stack CP^ acts on the fiber 
M„ of hd^^ over a containing a principal homogeneous space as an open substack. For 
generic a, "Pa acts simply transitively on Mq. In general, there exists a product formula 
that expresses the quotient [Ma/Ta] in terms of local data as in [N2l IN3j . 

Assume from now on that G is semisimple. As in [CLj . there exists a open substack 
M^*^ of Md^fj, that is proper over Ad.^- This open substack depends on the choice of 
a stability condition. However, its cohomology should be independent of this choice. 
Moreover, there exists an open subset A^'^/^ of Ad^^ whose fc-points are the pairs {D, b) 
such that as an element of (T/W){F k), b corresponds to a regular semisimple and 
anisotropic conjugacy class in G{F 0k k). The preimage M^'^^ of ^1^°^ is contained in 
M^*^ for all stability conditions. In particular, the morphism M^"^ — > A'f^/^ is proper. 

Let T^J'jj^ be the open substack of CP^^^ such that for every a G Ad,^, the fiber CP*°^ is an 
open subgroup of Ta and the component group vro(CP*°'') is the torsion part of 7ro(J'a)- 
Whatever stability condition we choose, J")^"^ acts on M^*^. 
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Our goal is to understand the cohomology (14. 2p . According to formula ()2.34p , up to 
a shift and Tate twist (and replacing the local system L by £*), it is isomorphic to the 
dual of the cohomology with compact support of A^{JCd,p)- 

Conjecture 4.1. The restriction to the diagonal A^(3Crf^p) is a pure perverse sheaf. 

To compute the cohomology with compact support of A^{Xd,p), we consider the 
sheaf ih%^),AU^d,p) = {hf^).A*^{%d,p) on Af^ (recall that hf^ is proper). By 
Deligne's purity theorem, Conjecture 14.11 implies that (/i^*^)*A?^(IKrf^p) is a pure com- 
plex. Hence, geometrically, it is isomorphic to a direct sum of shifted simple perverse 
sheaves. 

As we explained in Section 13.51 we wish to compare the stable trace formulas for the 
given group G and the groups H (depending on p and d). Hence we need to isolate 
geometrically a part in the cohomology of (/i^*^)*A^(3Crf p) which corresponds to the 
stable trace. For this, we follow the strategy of [N2t IN3j . 

Let TToi'Pj'jj) denote the sheaf of connected components of ^If^. This is a sheaf of 
finite abelian groups for the etale topology of ^1^^^. As in |N3) . we define the stable part 

ihd,fj,)*'^ki^d,p)st 

as the largest direct factor of {hj^ ^)^A'^{'Xd,p) on which TTQ°'^(Pd,p) acts unipotently. 

Note that in the above formula the subscript "st" and the superscript "st" have 
completely different meaning: the subscript refers to the stable trace formula (see 
Section below) . and the superscript refers to imposing a stability condition in the 
sense of geometric invariant theory. 

By analogy with the main theorem of [N3 1 ICLj . we state the following conjecture. 

Conjecture 4.2. The support of any simple perverse constituent o/ (/i^*^)^, A^(3Crf p)st 
is In particular, for every integer n, the perverse sheaf 

PH-{{h%^).AU^d,p)st) 
is the intermediate extension of its restriction to any non-empty open subscheme of 

•^d,p- 

We will now explain the arithmetic meaning of this conjecture. 

4.3. Adelic description. Let M.d^^{D) be the fiber of M^^^ over D = Yll=i''^i[^i]- 
the curve X is defined over a finite field k, we can give an adelic expression for the 
groupoid M(i^^{D) of A:-points of M.d,p{D)- Namely, let 

MdAD) = {7, {g.).e\x\h e G{F),{g,) G G(A)/G(Oa), 5x Sffx € G(0,)A,(t,)G(0,)} 

C G{F) X G(A)/G(Oa), 

where A^. < dxP if x belongs to the support of D and A^^ = 0, otherwise. The group 
G{F) acts on Mii^^{D) by the formula 

h (7, {9x)) = {hjh^^, (/igx)xex) , 

and ^ 

Md,^iD) = G(F)\Mrf,p(Z)). 
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Indeed, {gx)xe\x\ determines a G-bundle E on X and 7 determines a section ip of Ad(£') 
on X — supp(L') satisfying conditions (|4.3p . 

The map /i^^^ sends a point (Z),7, (17^)) € Mrf^^(A;) to {D,b) G Ad,^{k) where b = 
(61, ... , 6,) with hi = tr(7) G Ox{{f^,uJi)D)). 

Let {D,b) G -^d'/il^)- According to the Lefschetz trace formula (see Section \T9\i . 
the trace of the Frobenius acting on the stalk of (/i^*^)*A^(3Crf_p) over {D,b) may be 
expressed as 

(4.7) Yl 0^i^d,p,D), 

7GG(F)/conj. 

where 7 runs over the set of conjugacy classes of G{F) that map to b. The global 
orbital integral O^^Kd^p^o) is defined by the formula 

0.y{Kd,p,D) = / Hd,p,D{g'^ig)dg 
JGj{f)\g{a) 

where the G.y{F) stabilizer of 7 in G{F) is a discrete subgroup of G(A) and dg is 
the Haar measure on G{A) such that G(Oa) has volume one. The function under the 
integration sign is 

Hd,p,D = ® ^Sym"-(-D)(p) 

xe\x\ 

on G(A), so that we have 

Kd,p,D{x,y)= ^ Hd,p,D{x~^ay) 
aeG(F) 

(see formula (j2.2p ). Here we write 

D= nx{D)[x], 
xe\x\ 

and Hsym"{p) is the element of the spherical Hecke algebra of G{Ox) bi-invariant func- 
tions on G{Fx) corresponding to the representation Sym"(/)) under the Satake isomor- 
phism. In particular, for all but finitely many x G \X\, H^^^n^iD)^^^ is the characteristic 
function of G{Ox)-, the unit element of the Hecke algebra. 

As in |N2] . for every {D,b) G A'f^l^{k) the trace of the Frobenius operator on the fiber 
of (/i^*^)*A^(3Cd^p)st over {D,b) can be expressed in terms of stable orbital integrals 

(4.8) n ^0,(i?sy^„.,z,)(p)). 

xG\X\ 

If {D,b) ^ yi^'^(A;) the expression is more complicated. Nevertheless, following [QLJ, 
it is reasonable to expect that it is the correct contribution of b to the Arthur stable 
trace formula. 
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4.4. The case of SL2. Let us consider again our example with G = SL2 and fi the 
highest weight of the adjoint representation of PGL2. As cocharacter of the maximal 
torus of SL2, jJL is simply t ^ diag(i,t~^). The moduli stack M^,^ classifies the data 

where D E is an effective divisor on X of degree d, E \s a, rank two vector bundle 
over X equipped with a trivialization of its determinant and is an automorphism of 
-^lx-supp(D) of determinant one that can be extended to a homomorphism of vector 
bundles 

tpi-.E-i- E{D). 

Let Ad^fj, be a fibration over X^, whose fiber over D G X^i is the vector space 
H^{X, Ox{D)). We have a Hitchin-Iikc map 

sending (D,E,ip) to {D,b), where 

fe = tr((^i) Ox (£>)). 

Using Riemann-Roch theorem, it is easy to see that 

dim(^rf,^) = 2d-g + l 

for every d > 2g — 1. This formula is in fact true for all d > g and we have a formula 
foi d < g as well. 

Recall that Aa^^, classifies ah pairs {D,b) with D e Xa and b e H^{X,Ox{D)). If 
6 is a non-zero section of H^{X,Ox{D)), the divisor D' = div(6) + D is an effective 
divisor of degree d that is linearly equivalent to D. Here div(6) is the principal divisor 
attached to 6 viewed an a non-zero rational function on X. Such a D' determines b up 
to a scalar in A;^. If denote the complement of the zero section 6 = 0, then we 



111 n'^ Tf 

have a map 



A^^^Xa X Xa 



given by {D,b) ^ {D,D') which is a G^-torsor. Here X/i — ^ Pic^ is the Abel-Jacobi 
map from X^ to the d-th component of Picard's variety (if we replace the Picard 
variety by the Picard stack, the above map is an isomorphism). In particular, we have 
the dimension formula 

dim(yid,^) = dim{Xd x Xd) + 1. 

Picd 

Now, we recall a classical formula from the theory of special divisors on compact Rie- 
mann surfaces. 

Lemma 4.3 (Martens). We have the following formula for the dimension of Xd x Xd 

Picd 

/. X N \d ifl<d<q — l, 

(4.9) dim(Xd X Xd) = \ i 

Pica [2d-g if g < d. 

Here g denotes the genus of X. 
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Proof. Suppose first 1 < d < g — 1. According to Martens' theorem [ACGHl p. 191], for 
every integer d such that 1 < d < g — 1, the Abel-Jacobi morphism Pic^ from 

Xd to the d-th component of he Picard variety is semi-smah over its image. In other 
words, 

dim{Xd X Xd) = d. 

Pica 

Suppose next g < d < 2g — 2. In this case, the map X^ Pic^ is surjective and 
its generic fiber is a projective space of dimension d — g. It follows that the irreducible 
component of X^ x X^ that dominates Pic^ has dimension 2d — g. It is enough to 

PiCd 

prove that the other components, if any, have less dimension. Let r be a positive integer 
and let denote by the locally closed subvariety of Pic^ consisting in line bundles L 
of degree d such that 

dimH^{X,L) = d+l-g + r. 

By Riemann-Roch theorem, dimH^(X, L') = r where L' = ® K where K is the 
canonical sheaf. We have deg(L') = d' = —d + 2g — 2. After Martens' theorem the 
subvariety of line bundles L' of degree d' such that d\m{H^{X, L')) = r is less or equal 
to d' — 2r + 2. It follows that the part of X^ x X^i over is of dimension at most 

PiCd 

2{d-g + r) + d' -2r + 2 = d, 

which is less than 2d — g. 

Finally, if d > 2g - 1, then for all D G Xd, dimi?°(X, Ox{D)) = d+l- g after 
Riemann-Roch theorem. It follows that the map X^ Pic^ is a projective bundle of 
rank d — g and the total dimension of X^ x X^ is 2d — g. □ 

PiCd 

We can use Hitchin's device called spectral curve to describe the fibers of hd^^- Recall 
that for any sections bi € H^{X,Ox{D)) and &2 £ H^{0x{2D)), we have a curve on 
the total space of the line bundle Ox(-D) defined by the equation 

- bit + 62 = 0. 

For every b G H^{X, Ox{D)^ l^t Ydj, be the spectral curve corresponding to particular 
parameters bi = b and 62 = I2D the constant function 1 on X considered as a global 
section of the line bundle Ox{2D). The map ip : E —?■ E{D) defines the structure of a 
module over the symmetric Ox-algebra of Qx{—D) on i?, so that E can be seen as an 
0-module on the total space of Ox{D). The Cayley-Hamilton theorem implies that E 
is supported on the spectral curve yo^ft. 
For every (L>, b) G .A^^^, the fiber 

classifies Oy^ j^-modules 3" such that by pushing along the finite fiat map pD,b '■ YD,b — > 
X, we get a rank two locally free Ox-module PD,b*-^' equipped with a trivialization of 
the determinant. If h is reduced, the fact that PD,b*^ ^ rank two vector bundle 
implies that 3" is a torsion-free module of generic rank one and vice versa. If Yd, 6 is 
smooth, 3" is in fact an invertible sheaf. 
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In the present SL2 case, the group "Pofi consists of invertible sheaves L on Y^^b with 
the trivial norm down to X. It acts on M/j^;, by tensor product 

because 

For b G H^{X,(Dx{D)) such that the section 6^ _ of the hne bundle Oxi^D) 
has a multiplicity free divisor, the spectral curve Yd,!) is smooth. In this case, yD,b is 
isomorphic to the quotient of an abelian variety by the trivial action of Z/2Z. In partic- 
ular, 'J'D,b is proper and connected. It is also known that '?D,b acts simply transitively 
on "M^D^b in this case. 

For a general parameter {D,b), as in [N3j, we need to keep track of two invariants 
attached to 'J'D,b- the component group '/ro(y_D,;,) and its invariant 6. 

The component group ttqCP D,b) is related to the theory of endoscopy. In the present 
situation, it is fairly easy to compute. If the spectral curve Yj^ b has at least one uni- 
branch ramification point, in particular irreducible, then 7ro(J'D,b) = 0. If this curve has 
only ramification points with two branches but still irreducible, then TTo{T£)^b) = Z/2Z. 
This is the case if and only if the normalization of Yu b is an unramified covering of X. 
Finally, if the spectral curve Yd, 6 has two irreducible components, then TToiTD^b) = ^• 
See the last section of |N2] for this calculation. 

The 5 invariant S{D, b) which is defined as the dimension of the affine part of the 
group I'd, 6 is a rough but efficient measure of singularity of "M^^b- This invariant can 
be calculated from the discriminant as follows. Let discr(Z), b) be the effective divisor 
attached to the section b^ — A2D of Ox {2D). In other words, we have 

discr(L>, b) = div(&2 _ 42/?) + 2D 

where div(5^ — 42_d) is the (virtual) divisor attached to the rational function 6^ — 42,0. 
Let us write discr(D, b) in the form discr(L', b) = Ylxex ''^xi^]- Then 

(4.10) mb) = Y,[^' 

where [^] denotes the integer part of A convenient way to express 5{D,b) from 
discr(D,6) is as follows : we write discr(L',6) under the form Di +2D2 where Di,D2 
are effective divisors and Di is multiplicity free. Then 

5(L»,6) =deg(L»2). 
We will stratify A^^^ by this invariant 6 

5 

where Ad^ij_{5) is the locally closed subscheme of ^l^^^ consisting of points {D,b) such 
that 5{D,b) = 6. In view of some general results proved in jN3j, the following estimate 
of dimensions provides some evidence in favor of Conjecture 14.21 

Proposition 4.4. Assume d > 2g — 1 and 6 < d — 2g + 1. Then the codimension in 
Ad,jx of the stratum Ad^j_t{5) is equal to 6. 
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Proof. For every integer e < d, let denote A(i,ij,{S,e) the locally closed subscheme of 
'^d,n{S) defined by the condition d{D,b) = S and the degree of the zero divisor of the 
rational function b equals e. Let {D, b) be a geometric point of Ad, ^{6, e). Let us denote 
Ej^ = div(6)+ the divisor of zeros of b and E- = div(£^)_ the divisor of poles; we have 
then deg(£'+) = dcg(-E_) = e. The requirement that the div(6) + £) be effective, implies 
that D = E- + Do for a certain effective divisor Dq of degree deg(Do) = = d — e. 
Consider the discriminant divisor 

discr(D, b) = div(62 - 42d) + 2L» 

where 

div(62 - 42d) = div(6^ - ^20)+ - 2^- 
since div(6^ — ^20)- = div(6^) and therefore 

discr(Z), b) = div(6 - 2^)+ + div(6 + 2d)+ + IDq. 

We observe that b defines a finite flat morphism 6 : X — t- of degree e and in terms of 
this map div(fe — 2/))_ = b~^{2) and div(6 + 2) = b~^{—2). In particular, these effective 
divisors are disjoint. 

Now we write div(6 - 2d)+ = D[ + 2D'2 and div(6 + 2d)+ = D'{ + 2D'^ where 
D[ and D'( are multiplicity free effective divisors. We will use the obvious notation 
d[ = deg(L'^), ^2 = deg(L'2) ^^c. Then we have 

(5 = 4 + 4' + do 

since D[ + D'^ is also a multiplicity free divisor. 
We consider the map from Ad^^ld, e) to 

Xd[ X Xd'^ X Xd- X Xd'^ X Xdo- 

This is a G„t-torsor over the closed subset of the above product defined by only one 
condition: D'^ + 2D'2 and D'l + 2D2 are linearly equivalent. Now this subvariety of the 
quintuple product is smooth of dimension 

d[ + d'2 + d'{ + d2 + do- g = 2d- g-S 

if at least one of the integers d'i,d'2,d'{,d2 is larger or equal to 2^ — 1. We prove that 
either d[ or d" is larger than equal to 2^ — L 
Recall the equalities 

d[ + 24 + do = d'i + 24 + dQ = d 

and 

(5 = 4 + 4 + "fo- 
under the assumption d — 5 >2g — l,we can proceeds as follows: assume 4 < 4' then 
24 + do <5 so that the inequality 

4 > d - (5 > 25 - 1 

is satisfied. It follows that the dimension of Ad,ij,{5, e) \s 2d — g + 1 — 6. □ 
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4.5. The case of the twisted torus. Now we consider a non-split example. Namely, 
let H be the one-dimensional torus over X attached to a /X2-torsor £,h : X' X as 
in Section 12. 4[ The dual group of H is = Gm x T where F acts non-trivially on 
Gm through the quotient corresponding to £jy. We have defined a homomorphism 

— )• PGL2 X r in (j2.28p . We will consider the restriction of the representation 
p = p^x of PGL2 X r to ^H, where p^ is the adjoint representation of PGL2 and p^^ 
is the representation of T corresponding to The three-dimensional representation 
Ph breaks into a direct sum of a one-dimensional and a two-dimensional representations: 

PH = PO® Pi- 

We have seen earlier that the perverse sheaf 'Xd,p also breaks into a direct sum 

(4.11) Xd,pH = Qi\{BnnH xX^, x X^J. 

do+di=d 

The relevant moduli stack M^j^j, which we will denote here by M^ p^, is a disjoint 
union 

(4.12) ^d,pH= U Md,p„ido,di), 

do-\-di =d 

where '^d^pnido^di) classifies the data {V ^Dq^Di^O)^ where 

(L',Z)o,^i) GBun^xXdo x X'^^ 

and is an isomorphism of line bundles 

e : L{Di-t{Di)) L 

such that t{9)9 = 1. Recall that r is the involution on X' such that X' /t = X. We 
derive from 6 a symmetric isomorphism Ox'(^i) ~^ 'T*i^ X'iDi)) or, equivalently a 
descent datum of the line bundle Ox'iDi) to X. This defines a point in the cartesian 
product X'^^ Xpicd^(X') Picdi/2(-'^)- Lst us denote 

■^d,pHido,di) = Xdo X {X'a^ Xpicd^{A-') Picdi/2(^))- 
This space is empty for odd integers di. The Hitchin base is then the disjoint union 

■^d,pH = |_| ■^d,pH{do,di). 

d()^di=d 

The Hitchin map, restricted to each component J^d,pH{do,di), is just the projection 

^d,p„ido,di) = BuuH ^■A.d,pjj{do,di) Ad,p„{do,di) 
that has relative dimension dim(Bunj7) = g — 1. 

4.6. Conjecture. We recall that our goal is to analyze the (stable parts) of the coho- 
mologies of the restrictions of the sheaves Xd,p and 'Xd,pH ^d,p and M^ p^, respec- 
tively, and to use them to establish the desired identities (I3.12p . We return to our main 
example (see Section [3^ : G = SL2, p = p^^ L, where is the adjoint representation 
of = PGL2 and L is an order two local system on X, and H is the twisted torus 
over X corresponding to L. We state the following conjecture in this case. 
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Recall that 7ro(Bun//) = TLj^'L and let us denote by Bun^ the neutral component of 
Bun//. Let us also denote by r the involution of Ad^p^j given by 

r(L',A,I)i,^) = (L',Z)o,T(I)i),r(0)). 

This involution acts on the cohomology of A^^p^ (with compact support) and we will 
denote by the upper script r the space of invariants of this action. 

Conjecture 4.5. For sufficiently large d there exists a quasi-isomorphism between 
complexes of l-adic vector spaces equipped with Frobenius operators 

modulo the contribution of the Eisenstein series mentioned at the end of the Section \3.(A 

We remark that the vector space on the right hand side may be viewed as a subspace 
of RTc{^d,pH-: 'Q^)' since ^d,pH — Bun// xAd^pjj- Taking r-invariants is due to the fact 
that the map Ad^pjj — > Ad^p is generically two-to-one, and taking the neutral component 
Bun^/ is the geometric counterpart of the factor 1/2 in the corresponding stable trace 
formula. 

We hope that this conjecture may provide a starting point for further investigation 
of trace formulas by geometric methods. 
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Part II 



In this, more speculative, part of the paper we propose two conjectural "geometric 
trace formulas" in the case of a complex algebraic curve X. 

5. The geometric trace formula 

In Section 12.91 we have interpreted geometrically the right hand side of the trace 
formula (jl.ip . Now we turn to the left hand side. From now on we will assume that G 
is a constant group scheme G x X, where G is a split reductive group defined over the 
ground field k. Henceforth, in order to simplify our notation, we will not distinguish 
between G and G. 



5.1. The left hand side of the trace formula. As we discussed in Section [3131 the 
L-packets of (unramified) irreducible automorphic representations should correspond to 
(unramified) homomorphisms Wp x SL2 ^G. Assuming this conjecture and ignoring 
for the moment the contribution of the continuous spectrum, we may write the left hand 
side of (O) as 



(5.1) TVK = ^m<,Ar,, 

a 

where a runs over the unramified homomorphisms Wp x SL2 — )• ^G, m^j is the multi- 
plicity of the irreducible automorphic representation unramified with respect to G{0) 
in the L-packet corresponding to a, and is the eigenvalue of the operator K on an 
unramified automorphic function on BunG'(Fg) corresponding to a spherical vector 
in this representation: 

K • = N„U. 
Thus, recalling ()2.3p . the trace formula ()2.ip becomes 

(5.2) -"^-^ E iA^aoi^^'*^-^)- 

Consider, for example, the case of K = Hp^^;, the Hecke operator corresponding to a 
representation p of ^G and x € \X\. Then, according to the conjectures of Langlands 
and Arthur, 

(5.3) N, = Ti\a\\''l 1 xFr.- 1 >P 




The operators K introduced in Section [2.81 are generated by the Hecke operators tip^x- 
Therefore the eigenvalue for such an operator K is expressed in terms of the traces 
of aiYix) on representations of ^G. 



42 



EDWARD FRENKEL AND NGO BAG CHAU 



5.2. Lefschetz fixed point formula interpretation. It is tempting to try to inter- 
pret the left hand side of (j5.2p as coming from the Lefschetz trace formula for the trace 
of the Frobenius on the cohomology of an ^-adic sheaf on a moduli stack, whose set 
of fc-points is the set of u's. Unfortunately, such a stack does not exist if fc is a finite 
field ¥q (or its algebraic closure). On the other hand, if X is over C, then there is an 
algebraic stack Loclq of (de Rham) ^G-local systems on X; that is, ■^G-bundles on X 
with flat connection. But in this case there is no Frobenius acting on the cohomology 
whose trace would yield the desired number (the left hand side of (j5.2p ). Nevertheless, 
we will define a certain vector space (when X is over C), which we will declare to be 
a geometric avatar of the left hand side of (|5.2p (we will give an heuristic explanation 
for this in Section [6. 4p . We will then conjecture that this space is isomorphic to (|2.36p 
- this will be the statement of the "geometric trace formula" that we propose in this 
paper. 

Let us first consider the simplest case of the Hecke operator K = Hp^x- In this 
case the eigenvalue N^- is given by formula (j5.3p . which is essentially the trace of 
the Frobenius of x on the vector space which is the stalk of the local system on X 
corresponding to a and p. These vector spaces are fibers of a natural vector bundle on 
X X hocLQ (when X is defined over C). 

Indeed, we have a tautological ^G-bundle T on X x Loclq, whose restriction to X xa 
is the ^G-bundle on X underlying a € Locl^;. For a representation p of ^G, let Tp be 
the associated vector bundle on X x IjOClq. It then has a partial flat connection along 
X. Further, for each point x \X\, we denote by T^. and Ip^^ the restrictions of T and 
7p, respectively, to x x Locl,^. 

It is tempting to say that the geometric incarnation of the left hand side of ()5.2p in 
the case K = Hp^x is the cohomology 

H*{LocLG,7p^x)- 

However, this would only make sense if the vector bundle 7p_x carried a flat connection 
(i.e., a D-module structure) and this cohomology was understood as the de Rham 
cohomology (which is the analogue of the etale cohomology of an £-adic sheaf that we 
now wish to imitate when X is deflned over C). 

Unfortunately, 7p^x does not carry any natural connection. If we have a vector bundle 
V with a fiat connection on a smooth algebraic variety Y, then its de Rham cohomology 
may be defined as the coherent cohomology of the tensor product 

V(g)A'(ry), 

where TY is the tangent sheaf to Y. This cohomology is well-defined even if V does not 
have a fiat connection, and so, by making a leap of faith, we can declare the cohomology 

(5.4) if(LocLG,Tp,,0A'(rLocLG)) 

as the geometric incarnation of the left hand side of (|5.2p . 

However, this formula is only the first approximation to the right formula, because 
LiOClq is not a variety, but an algebraic stack. We suggest that the correct formula is 

RHom(A,(0),A,(Tp,^)) =RHom(A*A,(0),Tp,^), 



GEOMETRIZATION OF TRACE FORMULAS 



43 



where A is the diagonal. This will be discussed further in Sections 16.31 and 16.41 

On the other hand, as explained in Section 12.91 the geometric avatar of the right 
hand side of (j5.2p is the (de Rham) cohomology 

H'iBunc, A- 

The conjectural geometric trace formula for K = Mp^x would then be an isomorphism 

(5.5) RHom(A*A,(0),Tp,,) ~ ii"(BunG, A'(DCp,,)). 

5.3. The case of K = K^^p. Now we generalize this to other integral operators K, in 
particular, the one that is of most interest to us, K = K^p (see formula ()2.26p ). In 
this case N^j on the left hand side of the trace formula ()5.2p is given by the formula 

E n Tr f f %] X Fr..) , Sym- {p)] . 

The corresponding sheaf on Locl^ is defined by symmetrization of the vector bundles 
Tp, as follows. 

First, let us recall from Section [2.61 that if is a flat vector bundle on X, then we 
define its dth symmetric power Ed by the formula 

(5.6) Ed={4{E^'^)Y\ 

where vr*^ : X'^ is the symmetrization map. This is a D-module on X^ (but not a 

vector bundle if the rank of E is greater than 1). We may perform the same construction 
over any base B parametrizing a family of flat vector bundles. In other words, suppose 
that we have a vector bundle £ over X x B equipped with a flat connection along X. 
Then we obtain a coherent sheaf 8,d on X^ x B which is moreover a D-module along 
Xd- The restriction of 8,d to Xd x b is Ed given by formula (15. 6p . where E = £|xxfe- 

Let us apply this in the case of B = Loclq. If p is again a representation of ^G, 
we have a tautological vector bundle Tp on X x Locl^j equipped with a flat connection 
along X. Let be the sheaf on X'^ x Loclq obtained by taking the Cartesian power 
of Tp along X. It carries a flat connection along X'^. Denote by vf'^ the symmetrization 
morphism X'^ x Loclq — > Xd x Loclq along the first factor and set 

This is a coherent sheaf on Xd x Loclq which carries the structure of a D-module along 
Xd- For each effective divisor D = nj[xj] on X, the restriction of T^^p to D x Locl^j 

is the vector bundle ^^T^J^x}, where Tp"2. is the tautological vector bundle on Loci,,^ 
corresponding to the representation Sym"'(/9) and Xj G \X\. 

Let TT : Xd X Loclq Locl^ be the projection onto the second factor. Then 

(5.7) 3-rf,p = m(Td,p) 

(where the direct image corresponds to taking the fiberwise de Rham cohomology) is 
the coherent sheaf on Locl^ that captures the functor IK = IK^^p the way the vector 
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bundle 7p^x captures the Hecke functor K = M.p^x- Hence, following the same argument 
as in the case of K = IHp^x, we may view the vector space 

(5.8) ii''(LocLG,3'd,p<^ A'(rLocLG)) 

as the first approximation to the geometric avatar of the left hand side of the trace 
formula (j5.2p for K = K^^p (it is approximate because LocL^r is an algebraic stack). 

On the other hand, as explained in Section 12.91 the geometric avatar of the right 
hand side of (j5.2p is the (de Rham) cohomology 

//•(Buna,A'(5Cd,p)). 

Therefore, in the first approximation, we obtain the following conjectural geometriza- 
tion of the trace formula ()5.2p : 

(5.9) i?'(LocLG,^rf,P® A'(TLociG)) -^•(BunG,A'(5Cd,p)). 

However, since Loclq is an algebraic stack in general, the left hand side of (15.91) 
has to be modified. In the next section we will argue that (j5.9p should be replaced 
by the isomorphism (16.91) which we will propose as the sought-after "geomeric trace 
formula" . We will also show that this isomorphism naturally appears as a corollary of 
the categorical Langlands correspondence (also known as "non-abelian Fourier-Mukai 
transform" ) . 

6. Categorical Langlands correspondence 

According to the geometric Langlands conjecture |BDj (see, e.g., jFlj for an exposi- 
tion), for each (sufficiently generic) ^G-local system a on X there exists a D-module 
(or perverse sheaf) 3'o- on Bunc which is a Hecke eigensheaf with "eigenvalue" a. This 
property means that we have a system of isomorphisms 

(6.1) ]Hp(?-,) ~ (po a)^?-, 

compatible with the tensor product structures, where Hp is the Hecke functor associated 
to the representation p of ^G, acting from the category of D-modules on Bunc to the 
category of D-modules on X x Bun^ (see Section 12. 2p . 

6.1. Equivalence of categories. Categorical Langlands correspondence extends this 
to a kind of spectral decomposition of the derived category of D-modules on Bunc. 
More precisely, one hopes that there exists an equivalence of derived categorie^ 



derived category of 
D-modules on Buug 



^ It is expected (see [FW] . Sect. 10) that in general there is not a canonical equivalence, but rather 
a Z2-gerbe of such equivalences. This gerbe is trivial, but not canonically trivialized. One gets a 
particular trivialization of this gerbe, and hence a particular equivalence C, for each choice of the 
square root of the canonical line bundle Kx on X. 



(6.2) 



derived category of 
0-modules on Locl^j 
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under which the skyscraper coherent sheaf supported at u E Loclq goes to 3"o-- This 
has been proved in the abehan case by Laumon [Laul] and Rothstein [Rj, as a version of 
the Fourier-Mukai transform. In the non-abehan case this categorical correspondence 
("non-abehan Fourier-Mukai transform") has been suggested by Beihnson and Drinfeld 
(see, e.g., |FH IVLa^ ). It also follows from the S-duality /Mirror Symmetry picture of 
Kapustin and Witten [KW] (see \FW\ IF2j for an exposition). It has not yet been made 
into a precise conjecture in the literature, so we only use it as a guiding principled 

In what follows, we will denote by -D(Loclq) and -D(BunG) the categories that should 
appear on the two sides of (j6.2p . 

This conjectural equivalence has been mostly studied at the level of objects (for 
instance, in the case when C is applied to the skyscraper sheaf 0^ on Loclq, see 
the next subsection). But it should also yield important information at the level of 
morphisms. In particular, if we denote the equivalence going from left to right in the 
above diagram by C, then we should have isomorphisms 

(6.3) RHom^(Lo,^^)(Ji,5-2) ^ RHomB(BunG)(C(^i), ^^(^2)) 
and 

(6.4) RHom(Fi,F2) ~RHom(C(Fi),C(F2)), 

where Fi and F2 are arbitrary two functors from the category D(Loclq) to itself. We 
will use the latter to produce the geometric trace formula (16. 9p . 

6.2. Compatibility with Wilson/Hecke functors. There is also another important 
property that the equivalence C is expected to satisfy. On both sides we have natural 
functors labeled by representations p of ^G. On the right hand side these are the 
Hecke functors Hp acting from the category of D-modules on Bunc to the category 
of D-modules on X x Bung (see Section 12. 2p . On the left hand side these are the 
functors acting from the category of 0-modules on Loclq to the category of sheaves 
on X X Loclq, which are D-modules along X and 0-modules along Loclq. Following 
|KW| . we will call them Wilson functors and denote them by Wp. 
By definition, 

(6.5) Wp{^) = 7p(g>p*^{J), 

where 7p is the tautological vector bundle on X x Loclq defined in Section 15.21 and 
P2 : X X Loclq — >■ Loclq is the natural projection. Note that, by construction, 7p 
carries a connection along X and so the right hand side of (j6.5p is a D-module along 
X. The Wilson functor Wp may also be written as the "integral transform" functor 
corresponding to the 0-module A*('Jp) on X x Loclq x Loclq, where A is the diagonal 
embedding of X x Locl,^: 

j^g*((?'*(:j)»A,(Tp))), 

where q and q' are the projections onto X x Loclq (the first factor) and Locl^ (the 
second factor), respectively. 



D. Arinkin, V. Drinfeld, and D. Gaitsgory have announced in recent talks a more precise formulation 
of this conjecture. 
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We may also consider more general functors built from the Wilson functors - for 
example, the functor Wd,p defined by the formula 

where 3'd,p is an 0-module on Locl^ given by formula (j5.7p . 

The compatibility of the equivalence C with the Wilson/Hecke functors is the state- 
ment that we should have a family of isomorphisms, which are compatible with the 
tensor product structures on the Wilson and Hecke functors, 

(6.6) C{Wp) ~ Mp, pe Rep(^G). 
This implies that we have functorial isomorphisms 

(6.7) C(Wp(:J)) ~ Mp(C(J)), peRep(^G). 

In particular, observe that the skyscraper sheaf Oq- at a € IjOClq is obviously an 
eigensheaf of the Wilson functors: 

Wp(0,) = (po a)MO^. 

Indeed, if we tensor a skyscraper sheaf with a vector bundle, this is the same as ten- 
soring it with the fiber of this vector bundle at the point of support of this skyscraper 
sheaf. Therefore ()6.7p implies that C(Oo-) must be a Hecke eigensheaf on Bunc with 
"eigenvalue" a (see formula (j6.ip ). Hence we recover the traditional formulation of the 
geometric Langlands correspondence. 

6.3. Application to the geometric trace formula. Consider now the isomorphism 
()6.4p in the case Fi = Id and F2 = Wd,p- It follows from the construction of the functors 
Wd,p and Md^p and formulas (|6.6p and ()6.7p that 

C{Wd,p) ^ Kd,p. 

Hence (|6.4p gives us an isomorphism 

RHom(Id,Wd,p) ~ RHom(Id,]Kd,p). 

We should also have an isomorphism of the RHoms's of the kernels defining these 
functors. On the right hand side this is the RHom 

RHom(A!(e),3Crf,p) 

(where S is the constant sheaf on Bunc and %d,p = P*(3^d,p)) in the derived category of 
D-modules on Bunc x Bunc, which we have proposed in formula (j2.32p as a geometric 
analogue of the right hand side of the trace formula. 

On the left hand side of the categorical Langlands correspondence, the kernel cor- 
responding to Wd,p is just A^{3'd^p), where is the 0-module on Locl^ defined by 
(j5.7p . In other words, it is supported on the diagonal in Locl^t x Locl^j and is just the 
0-module push-forward of 3'd^p from the diagonal. 

Thus, we find that the categorical version of the geometric Langlands correspondence 
should yield the following isomorphism 



(6.8) 



RHom(A,(0), A,(:Jrf,p)) ~ RHom(A!(e), 3^^,^). 



GEOMETRIZATION OF TRACE FORMULAS 47 

We recall from Section 12.91 that the right hand side is isomorphic to 

We may also rewrite the left hand side of (|6.8p using adjunction as follows: 

RHom(A*A,(0), 

Hence we obtain the following isomorphism, which we conjecture as a geometric trace 
formula. 

Conjecture 6.1. There is an isomorphism of vector spaces: 

(6.9) RHom(A*A,(0), Jd,p) ~ H*{BnnG, A-{%d,p)). 

Thus, starting with the categorical Langlands correspondence, we have arrived at 
what we propose as a geometrization of the trace formula. 

We have a similar conjecture for more general functors IC built from the Hecke 
functors of the type introduced in Section 12.81 On the other side of the categorical 
Langlands correspondence it corresponds to a functor W built in the same way from 
the Wilson functors, so that 

C{K) ~ W. 

We then expect to have an analogous isomorphism 

(6.10) RHom(A*A,(0),3") ~ F'(BunG, A '(%)). 

where % is the kernel of IK and 3" is the 0-module on Loclq such that A^,(3") is the 
kernel of W. 

Remark 1. It is tempting to rewrite the left hand side of (j6.10p as 

RHom(0, A- A*(3")) = H'(Loclg, A'A*(3")), 
where A' is the right adjoint functor to A*. Then (j6.10p becomes more symmetrical: 

(6.11) -H''(Loclg,A'A,(3")) ~ //•(BunG,A'(5C)). 

However, the definition of the functor A' for the derived category of coherent sheaves 
on the stack Loclq is tricky. Defining the left hand side of (16. lip essentially amounts 
to rewriting it as the left hand side of (j6.10p . That is why we stated the conjecture in 
the form (l6T0]l . □ 

As a special case, let both K and W be the identity functors. Then ()6.10p gives rise 
to an isomorphism 

(6.12) RHom(A*A,(0), 0) ~ H^^iBunc). 

The right hand side of this formula is known [TJ INSl |HS] , and hence this special case 
already gives us a good test for the conjecture. The isomorphism (|6.12p holds in the 
abelian case G = Gm, as we will see at the end of in Section! 



48 



EDWARD FRENKEL AND NGO BAG CHAU 



6.4. Connection to the Atiyah Bott-Lefschetz fixed point formula. Here we 
give an heuristic explanation why we should think of the space (I6.10p as a geometric 
incarnation of the sum appearing on the left hand side of (15. 2p . 

Let 9" be a coherent sheaf on Loclq built from the vector bundles 7p^x,x G |X| (like 
3'd,p constructed below). We would like to interpret taking the trace of the Frobenius 
on the (coherent!) cohomology H'(hocLQ, 3") as the sum over points a of Loclq, which 
we think of as the fixed points of the Frobenius automorphism acting on a moduli of 
homomorphisms 

k 

Recall the Atiyah-Bott-Lefshetz fixed point formula |AB] (see also §6). Let M 
be a smooth proper scheme, V a vector bundle on M, and V the coherent sheaf of 
sections of V . Let u be an automorphism acting on M with isolated fixed points, and 
suppose that we have an isomorphism 7 : u*(y) ~ V. Then 

(6-13) TV(7,/7'(M,V)) = y ^^^^^'^P^ 

^ ' v/> V > 7; ^ det 1 - 7, T*M ' 



where 



M" = r„ X A 

MxM 



is the set of fixed points of u, the fiber product of the graph r„ of u and the diagonal 
A in M X M, which we assume to be transversal to each other (as always, the left hand 
side of (I6.13P stands for the alternating sum of traces on the cohomologies). 

Now, if we take the cohomology not of V, but of the tensor product 'V®rt'{M)^ where 
ri*(M) = A*(T*(M)) is the graded space of differential forms, then the determinants 
in the denominators on the right hand side of formula (I6.13P will get cancelled and we 
will obtain the following formula: 

(6.14) Tr(7,if(M,V(g)17'(M))) = ^Tr(7,Vp). 

We would like to apply formula ()6.14p in our situation. However, \jOClq is not a 
scheme, but an algebraic stack (unless is a torus), so we need an analogue of (16.14p 
for algebraic stacks (and more generally, for derived algebraic stacks, since Locl^ is 
not smooth as an ordinary stack). 

Let M be as above and A : M the diagonal embedding. Observe that 

(6.15) A*A,(V) ~ V0f)'(M), 
and hence we can rewrite ()6.14p as follows: 

(6.16) Tr(7,F'(Af,A*A,(V)))= Tr(7,Vp). 

Now we propose formula (I6.16P as a conjectural generalization of the Atiyah-Bott- 
Lefschetz fixed point formula to the case that M is a smooth algebraic stack, and more 
generally, smooth derived algebraic stack (provided that both sides are well-defined). 
We may also allow here V to be a perfect complex (as in [H]). 
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Even more generally, we drop the assumption that u has fixed points (or that the 
graph Tu of u and the diagonal A are transversal) and conjecture the following general 
fixed point formula of Atiyah-Bott type for (derived) algebraic stacks. 

Conjecture 6.2. 

(6.17) Tr(7,/7'(Af, A*A,(V))) = Tiij, H' (M^ilA^iV))), 

where M" = x A is the fixed locus of u and iu : M" — ^ M x M. 

A/xM 

We want to apply ()6.16p to the left hand side of (|6.10p . which is 

RHom(A*A*(0), J). 

This is not exactly in the form of the left hand side of (j6.16p . but it is very close. 
Indeed, if M is a smooth scheme, then 

RHom(A*A*(0), J) ~ 9" (g) A* (TM) , 

so we obtain the exterior algebra of the tangent bundle instead of the exterior algebra 
of the cotangent bundle. In our setting, we want 7 to be the Frobenius, and so a fixed 
point is a homomorphism vri(X) ^G. The tangent space to a (in the derived sense) 
should then be identified with the cohomology H'{X,adoa)[l], and the cotangent 
space with its dual. Hence the trace of the Frobenius on the exterior algebra of the 
tangent space at a is the L-function L{a, ad, s) evaluated at s = 0. Poincare duality 
implies that the trace of the Frobenius on the exterior algebra of the cotangent space 
at a is 

L(a,ad,l) = g"'^«L(o-, ad, 0), 

so the ratio between the traces on the exterioir algebras of the tangent and cotangent 
bundles at the fixed points results in an overall factor which is a power of gE3 

Hence, by following this argument and switching from C to Fg, we obtain (up to a 
power of q) the trace formula (j5.2p from the isomorphism (|6.10p . 



Remark 2. The sheaf A*A,,(Ojv/) is the structure sheaf of the self-intersection of the 
diagonal of a derived stack M. In [BNJ it is interpreted as the loop space of M. Hence 
the left hand side of our conjectural generalization (16.17P of the Atiyah-Bott-Lefschetz 
formula may be viewed as the cohomology of the pullback of V from M to its loop 
space. David Nadler has outlined to us a possible proof of (I6.16P using the methods of 
[BFNllBN] . □ 



The exact power of q depends on whether we choose the geometric or the arithmetic Frobenius 
here. We recall from Section \2 . 91 that the right hand side of the trace formula (|5.2p is equal to the trace 
of the Frobenius also up to an overall factor which is a power of q. Since this is an heuristic argument, 
we ignore these factors. 
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6.5. Discussion. The isomorphism (I6.10p is stiU tentative, because there are some 
umesolved issues in the definition of the two sides of the isomorphism (I6.10p . 

The first issue is the definition of the stack Loclq. Presumably, it should be de- 
fined not as a stack, but as a derived (or differential graded, or DG) stack. This should 
correspond to the inclusion of the non-tempered (or non-Ramanujan) automorphic rep- 
resentations, which should be parametrized, according to Arthur, by homomorphisms 

a -.Wf X SL2^ 

(for tempered representations, (y\sL2 is expected to be trivial). For instance, the trivial 
representation of G(A) corresponds to a that is trivial on Wp and is a principal SL2- 
embedding on the second factor. Recent results of V. Lafforgue |VLaf| show that one 
might be able to take such a into account by considering a derived version of the stack 
IjOClq. There are also additional insights pointing in the same directions coming from 
the study of S-duality in physics (see a brief discussion at the end of [F2j and references 
therein). 

The second issue is the precise definition of RHom's in the two categories involved in 
the Langlands correspondence (j6.2p . which appear on the two sides of the isomorphism 
(j6.10p . This is in fact the main question in finding the correct formulation of the 
categorical Langlands correspondence. 

Finally, as we have discussed in Part I, in order to become an effective tool, trace 
formulas have to be stabilized. Hence an analogue of stabilization needs to be worked 
out in the geometric setting as well. 

But suppose we could solve these problems and really make sense of ()6.10p as a 
geometric trace formula. What would we be able to learn from it? 

First of all, we would have a good framework for the geometric trace formula that 
works for curves over Fg and over C. Second, we could try to establish the isomorphism 
(I6.10p by using fixed-point formulas, for example. Here the description of G^-fixed 
points in the moduli space of stable flat bundles due to Hitchin |HH IH3] and Simpson 
[Si] could be useful. 

The main application we have in mind is to use the isomorphism ()6.10p to establish 
functoriality. Namely, we wish to express each side as a direct sum of two vectors spaces 
corresponding to the tempered (Ramanujan) and non-tempered (non-Ramanujan) rep- 
resentations, and then to decompose further the former as a direct sum over the groups 
H (see Section [3. 5p . In Section H] we made the first steps toward this goal for the right 
hand (orbital) side of (15. 4p . using an analogue of the Hitchin fibration. The fact that 
we now have the left hand (spectral) side of the isomorphism (I6.10p may help us un- 
derstand better how to make this decomposition. For instance, the locus of Gm-fixed 
points in the moduli space of stable ^G-local systems decomposes into the union of the 

= locus (where 4> is the Higgs field) and other loci with </> 7^ 0. It is possible that 
the former corresponds to the tempered part and the latter to the non-tempered part. 

7. Relative geometric trace formula 

In the previous section we discussed a geometrization of the trace formula (II. ip . We 
have seen in the previous section that a geometric analogue of this formula may be 
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constructed within the framework of the categorical form of the Langlands correspon- 
dence, as the statement that the RHom's of kernels of certain natural functors are iso- 
morphic. These kernels are sheaves on algebraic stacks over the squares Bunc x Buug 
and Loclq x Loclq. 

It is natural to ask what kind of statement we may obtain if we consider instead the 
RHom's of sheaves on the stacks Bunc and Locl^j themselves. 

In this section we will show that this way we obtain what may be viewed as a geo- 
metric analogue of the so-called relative trace formula. On the spectral side of this 
formula we also have a sum like (15. ip . but with one important modification; namely, 
the eigenvalues N^- are weighted with the factor L(a", ad, 1)~^. The insertion of this 
factor was originally suggested by Sarnak in [S] and further studied by Venkatesh |Ve] . 
for the group GL2 in the number field context. Another difference is that the summa- 
tion is restricted to the generic representations, which means (at least, conjecturally) 
that we remove the contribution of the non-Ramanujan representations as well as the 
multiplicity factors ruo-- 

We will also analyze the corresponding orbital side and express is as the cohomology 
of a sheaf on an algebraic stack. This will give us what we may call a relative geometric 
trace formula. It will be the statement about an isomorphism of two cohomology 
spaces, one on Bunc and the other on Locl^. We will see that this isomorphism arises 
naturally from the categorical form of the geometric Langlands correspondence (j6.2p . 

7.1. Relative trace formula. First, let us recall the setup of the relative trace for- 
mula. 

Let G be a split simple algebraic group over k = ¥q. In order to state the relative 
trace formula, we need to choose a non-degenerate character of N(F)\N(A), where N 
is a maximal unipotent subgroup of G. A convenient way to define it is to consider a 
twist of the group G. Let us pick a maximal torus T such that B = TN is a Borel 
subgroup. If the maximal torus T admits the cocharacter p : T equal to half-sum 

of all positive roots (corresponding to B), then let K'^ be the T-bundle on our curve 
X which is the pushout of the Gm-bundle (the canonical line bundle on X without 
the zero section) under p. If p is not a cocharacter of T, then its square is, and hence 

this T-bundle is well-defined for each choice of the square root K"^ of Kx. We will 
make that choice once and for allE!l 
Now set 

G^ = K^^x G, = K'Px N, 

where T acts via the adjoint action. For instance, if G = GL„, then GL^ is the group 
scheme of automorphism of the rank n bundle © Kx © ... © K^'^ on X (rather 
than the trivial bundle 0®"). 
We have 

(7.1) iV^/[iV^,iV^] = 



This choice is related to the ambiguity of the equivalence (|6.2[) . see the footnote on page 1441 
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Now let iIj :¥q ^ be an additive character, and define a character ^ of N^{A) as 
follows 

rank G 

^{{ux)x&\x\) = n n V'(TrA;^/feReSa;(u^,i)), 

x(^\X\ i=l 

where Ux,i € Kx{Fx) is the ith projection of Ux € N^{Fx) onto Kx{Fx) via the 
isomorphism (jT.ip . We denote by /c^ the residue field of x, which is a finite extension 
of the ground field A; = Fg. 

By the residue formula, ^ is trivial on the subgroup N^(F) (this was the reason 
why we introduced the twist). It is also trivial on N^{0). 

In what follows, in order to simplify notation, we will denote and simply by 
G and iV. 

Given an automorphic representation vr of G(A), we have the Whittaker functional 

W{f) = [ f{u)^-\u)du, 

Jn{f)\n(a) 

where du is the Haar measure on N{A) normalized so that the volume of A^(0) is equal 
to 1. 

We choose, for each unramified automorphic representation vr, a non-zero G(0)- 
invariant function /^r G vr on G{F)\G{A). 

Let K again be a kernel on the square of BunG'(A;) = G{F)\G{A)/G{0) and K 
the corresponding integral operator acting on unramified automorphic functions. The 
simplest unramified version of the relative trace formula reads (here we restrict the 
summation to cuspidal automorphic representations vr) 

(7.2) Y^W^iU) W^{K ■ = 

K {ui, U2)^^^ (ui)'^ {u2)duidu2, 



I 



N{F)\N{A) N{F)\N{A) 

(see, e.g., [J]), where 

\\f\\'= I \f{9)\'dg, 

GiF)\G{A) 

and dg denotes the invariant Haar measure normalized so that the volume of G{0) is 
equal to 1. Note that 

\\f\f = q''oL{G)\\f\\l 
where is the norm corresponding to the Tamagawa measure, dc = {g — 1) dimG, 

L{G) = wam, + i), 

i=l 

where the rrii are the exponents of G. 

The following conjecture was communicated to us by B. Gross and A. Ichino. In the 
case of G = SLn or PGLn, formula (|7.3p follows from the Rankin-Selberg convolution 
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formulas, as we explain in Section 17.21 below. Other cases have been considered in 
\GP\ ITcl llclk| . For a general semi-simple group G of adjoint type formula (j7.3p has 
been conjectured by A. Ichino and T. Ikeda assuming that tt is square-integrable. Note 
that if a square-integrable representation is tempered, then it is expected to be cuspidal, 
and that is why formula (17. 3p is stated only for cuspidal representations. 

Recall that an L-packet of automorphic representations is called generic if each 
irreducible representation tt = ^ tTx from this L-packet has the property that the 
local L-packet of tTx contains a generic representation (with respect to a particular 
choice of non-degenerate character of N{Fx)). 

Conjecture 7.1. Suppose that the L-packet corresponding to an unramified a : Wp — > 
is generic. Then it contains a unique, up to an isomorphism, irreducible represen- 
tation vr such that VFiii(/7r) ^ 0, with multiplicity m-,^ = 1. Moreover, if this vr is in 
addition cuspidal, then the following formula holds: 

(7.3) |VF*(^)|2||M|-2 = g'iiv-dG^(^^ad,l)-i|5,r\ 

where is the (finite) centralizer of the image of a in ^G, = —{g — l)(4(p, p) — 
dim A^) (see formula (j7.6p ), and dc = (g — 1) dimG. 

Furthermore, we expect that if the L-packet corresponding to a : Wp x SL2 — > ^G 
is non-generic, then 

(7.4) L(CT,ad,l)"^ = 0. 

Ichino has explained to us that according to Arthur's conjectures, square-integrable 
non-tempered representations are non-generic. 

Recall that we have K • /^r = Aq-Ztt. Therefore Conjecture 17.11 and formula (j7.2p give 
us the following: 

(7.5) q-''^ A^<x-L(a,ad,l)-Var' = 

q-'^'^ J j K{ui,U2)-^~^{ui)-^{u2)duidu2. 

On the left hand side we sum only over unramified o", and only those of them contribute 
for which the corresponding L-packet of automorphic representations vr is generic. 
We expect that the left hand side of formula (j7.5p has the following properties: 

(1) It does not include homomorphisms a : SL2 x Wp ^G which are non-trivial 
on the Artur's SL2. 

(2) The multiplicity factor m-o- of formula (15. 2p disappears, because only one irre- 
ducible representation from the L-packet corresponding to a shows up (with 
multiplicity one). 

(3) Since is the group of automorphisms of a, the factor jS'o-l"^ makes the sum 
on the left hand side (j7.5p look like the Lefschetz fixed point formula for stacks. 

7.2. The case of G = PGL„. Here we derive formula (|7.3p for G = PGLn using the 
Rankin-Selberg convolution method, following [Lysj . Denote by Bun„ the moduli stack 
of rank n bundles on X and Bun™ the connected component corresponding to vector 
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bundles of degree m. Let a be an irreducible n-dimensional £-adic representation of Wp 
and 3'o- the pure perverse sheaf of weight on Bun^ (irreducible on each Bun^) which 
is a Hecke eigensheaf with respect to a (see |FGV2] ). Let /o- be the corresponding 
function on HuiiGL^iFq). The following formula is derived in |Lys : 



X] 1^ ^^N| /a(-^)/a* (L) = Ress=i L{a, ad, s) ^ 

(here ad denotes the adjoint representation of GL„). In fact, S. Lysenko [Lysj has given 
a geometric interpretation of this formula, which is compatible with the categorical 
Langlands correspondence. 

In addition, for irreducible a we have 

where 

(g-l)n(n-l)(2n-l) / 1^ 2 

dn = 7, > dG = [g - l)n , 



see |LLj . 

Using this formula, it is straightforward to obtain for G = PGL„ that 

mifn)\^\\fnr^ = -q''''-''^L{a,ad,ir\ 
n 

where ad denotes the adjoint representation of = SL„. This agrees with formula 
(|7.3|) . because 5o- is the center of ^G in this case (here we are again under the assumption 
that a is irreducible). 

7.3. Geometric meaning: right hand side. Now we discuss the geometric meaning 
of formula (|7.5p . starting with the right hand side. Let Bun^ be the moduli stack of 
B = bundles on X such that the corresponding T-bundle is 3"t = K'^. Note that 

(7.6) dimBun^^ = dN = -{g - l){A{p, p) - dimiV). 
Let ev : Bun^^ — )• Gq be the map constructed in jFGVlj . 

For instance, if G = GL2, then Bun^ classifies rank two vector bundles V on X 
which fit in the exact sequence 

(7.7) ^ kH"^ ^ V ^ K^^'"^ 0, 
1/2 

where K-^ is a square root of Kx which we have fixed. The map ev assigns to such V its 
extension class in Ext(Ox, -^x) = H^{X, Kx) — Ga- For other groups the construction 
is similar (see |FGVlj ). 

On Ga we have the Artin-Schreier sheaf associated to the additive character ip. 
We define the sheaf 

$ = ev*(£|v,) 

on Bun^. Next, let p : Bun^^ Bun^ be the natural morphism. Let 

^ = p,.{^)[dN - dcWx - dG)/2). 
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Then the right hand side of ()7.5p is equal to the trace of the Frobenius on the vector 
space 

(7.8) RHom(*,IC(^')). 

Here we use the fact that B o K ~ K o B and ~ ev*{L\^-i)[2dN]{dN)- 

7.4. Geometric meaning: left hand side. As discussed above, we don't have an 
algebraic stack parametrizing homomorphisms a : Wp — >■ if our curve X is defined 
over a finite field ¥q. But such a stack exists when X is over C, though in this case 
there is no Frobenius operator on the cohomology whose trace would yield the desired 
number (the left hand side of (|7.5p ). In this subsection we will define a certain vector 
space (when X is over C) and conjecture that it is isomorphic to the vector space 
(j7.14p which is the geometric incarnation of the right hand side of (j7.5p (and which 
is well-defined for X over both and C). This will be our "relative geometric trace 
formula". In the next subsection we will show that this isomorphism is a corollary of 
the categorical version of the geometric Langlands correspondence. 

In order to define this vector space, we will use the coherent sheaf on Locl^ 
introduced in formula (15. 7p . We propose that the geometric incarnation of the left hand 
side of (17. 5 p in the case when K = IC^^p is the cohomology 

(7.9) H'{LocLG,3'd,p). 

The heuristic explanation for this proceeds along the lines of the explanation given in 
the case of the ordinary trace formula in Section 16.41 using the Atiyah-Bott-Lefschetz 
fixed point formulaEl We wish to apply it to the cohomology (17.9p . If Locl^ were a 
smooth scheme, then we would have to multiply the number N^^ which corresponds to 
the stalk of at a, by the factor 

(7.10) det(l-Fr,r;LocLG)"^- 

Recall that the tangent space to a (in the derived sense) may be identified with the 
cohomology H'{X,adoa)[l]. Using the Poincare duality, we find that the factor (I7.10p 
is equal to 

L{a, ad, 1)^^. 

Therefore, if we could apply the Lefschetz fixed point formula to the cohomology 
(17. 9|) and write it as a sum over all a : Wp ^G, then the result would be the left 
hand side of (|7.5p (up to a factor that is a power of q). (Note however that since 
Locl(5 is not a scheme, but an algebraic stack, the weighting factor should be more 
complicated for those a which admit non-trivial automorphisms, see the conjectural 
fixed point formula (16.17P in Section [6741 ) 

This leads us to the following relative geometric trace formula (in the case of the 
functor Kd^p). 

Conjecture 7.2. We have the following isomorphism of vector spaces: 

(7.11) H'{LocLG,3'd,p) - RHomBunG(*,IKrf,p(^))- 



We note that applications of the Atiyah-Bott-Lefschetz fixed point formula in the context of 
Galois representations have been previously considered by M. Kontsevich in [K] , 
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It is clear how to generahze this to other functors IfC of the form described in Sec- 
tion ESI 

Now we explain how the isomorphism (|7.1ip fits in the framework of a categorical 
version of the geometric Langlands correspondence. 

7.5. Interpretation from the point of view of the categorical Langlands cor- 
respondence. We start by asking what is the D-module on Bunc corresponding to 
the structure sheaf on Locl^; under the functor C discussed in Section [6l The fol- 
lowing answer was suggested by Drinfeld (see [VLaf] ) : it is the sheaf \& that we have 
discussed above. 

The rationale for this proposal is the following: we have 

RHomLoci^(0,Oa) = C, VcT, 

where Oo- is again the skyscraper sheaf supported at a. Therefore, since C(Oo-) = la-, 
we should have, according to ()6.3p . 

RHomBunG('^(0),5-,) = C, VcT. 

According to the conjecture of jLLj. the sheaf ^ has just this property: 

RHomBunG(^,^<7) 

is the one-dimensional vector space in cohomological degree (if we use appropriate 
normalization for 3"(j). 

This vector space should be viewed as a geometric incarnation of the Fourier coeffi- 
cient of the automorphic function corresponding to 3"o-- 

This provides some justification for the assertion thad 

(7.12) C{0) = *. 

Let us take this for granted. 

Now let us look at the left hand side of the isomorphism (17. lip . We can rewrite it 
as the de Rham cohomology of a sheaf on X^, whose fiber at = ^ ni[xi] € is 

(7.13) RHomLoc,^ =RHom (^0, W(;^.|(0)^ , 

where Wp"^] is the Wilson operator corresponding to the representation Sym"* p, spe- 



cialized to the point Xi ^ X (see formula ()6.5p ). 

Using the compatibility (16. 7p with the Wilson/Hecke operators and formulas (I6.3p 
and (I7.12p . we obtain that (I7.13P should be isomorphic to 

(7.14) RHomBunc , 

where EIp"^] is the Hecke operator corresponding to the representation Sym""' p, special- 
ized to the point Xi ^ X. Hence the right hand side of (j7.1ip is isomorphic to the de 
Rham cohomology of a sheaf on Xd whose fiber at D = ^ [xj\ G X^ is (I7.14p . But 



explained in the footnote on page 1441 C corresponds to a particular choice of K^/^ . Given such 



1/2 

a choice, C(0) should be the character sheaf '5 associated to that . 
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varying D over Xd gives us precisely our averaging functor K^^p. Therefore the result 
is 

(7.15) RHomBunG(^,IKd,p(^')), 

which is the right hand side of (j7.1ip . 

Thus, we obtain that the relative geometric trace formula ()7.1ip follows from the 
categorical version of the geometric Langlands correspondence. 

Let's recap. We started out with the classical relative trace formula ()7.5p . The right 
hand side of ()7.5p had a simple geometric interpretation as the trace of the Probenius 
on the vector space (|7.14p . To interpret the left hand side of (|7.5p we made a "leap 
of faith" and replaced it by the cohomology (j7.9p . However, we have just shown that 
the resulting geometric relative trace formula (j7.1ip is a meaningful statement from 
the point of view of the categorical geometric Langlands correspondence. This is an 
indication that (|7.1ip is actually a reasonable conjecture (and hence so is our "leap of 
faith"). 

7.6. Applications to functoriality. As explained in jFLNj (following \L2 \ IL3t lL4] ). 

our goal is to express the trace of the averaging operator p for the group G as the 
sum of traces for the groups H dual to subgroups C for which representation p 
contains non-zero invariant vectors. This has important applications to functoriality. 

Likewise, in the setting of the relative trace formula, we are interested in decomposing 
the left hand side as a sum over the groups H. It is instructive to see why and how this 
decomposition should come about, first for an arbitrary G and then in the example of 
G = GL2 and p the second symmetric power of its defining representation. 

Let us consider the general case first. Look at the left hand side of formula (17. 5p . 
According to Corollary 13.31 (see also [FLN] . Lemma 2.6), N^j = for d > {2g — 2) dim /), 
unless {p o fj) contains non-zero invariant vectors. From now on we will assume that 
d > {2g — 2) dim p. Then we obtain that the left hand side of formula (j7.5p decomposes 
into a sum 

(7.16) Yl Yl N^'-L{a',adLG,q-^r^\S^r^ 

over all possible C ^G such that has a trivial subrepresentation in po a. Then 
each a' : Wp — > gives rise to a : Wp ^G and to po a' : Wp — > Aut p (we view p 
as a representation of obtained by restriction from ^G). We have 

(7.17) N^, = Tr I^Fr, ^(-l)^F^(A,, {p o a')d)^ 

= q~'^'' -coeS . of L{a' , p, s) . 

Observe that = •^f) ® Rh, where ^f) is the Lie algebra of and Rh is a certain 
representation of ^H. We have 

L(cT, adig, 1) = L(a', adxj^, 1) • L{a' , Rh, 1). 
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Hence the term in the sum (I7.16P corresponding to a particular '^H reads 

(7.18) N^' ■L{a',a.d,.H^)~^ -LW^RhA)-^ ■\S^\-\ 

The difference between (jT.lSp and the left hand side of (I7.5P is that we have the addi- 
tional weighting factor 

(7.19) L{a',RH,l)-^ 

(note that it corresponds to the conormal bundle to Locaj:^ in Locl^). 
Thus, the formula which we want to prove is 

(7.20) YN„-L{a,a.d,l)-^\S^\^^ = 

The summation is over the described above. 

In the same way as for the ordinary trace formula (as explained in [ FLN| ). our goal 
is to obtain the sum decomposition (j7.20p using the right hand side of formula (j7.5p . 

Geometrically, we would like to have a decomposition of the right hand side of ()7.1ip 
as a direct sum of spaces labeled by the ^H. We will also need to construct the geometric 
counterpart of the extra factor (j7.19p . 

We now look more closely at what happens in one example. 

7.7. The case of GL2. Let p be the second symmetric power of the defining represen- 
tation of GL2. We want to decompose the vector space 

(7.21) RHomBunc^, (^',ItCd,p(^)), 

into a direct sum of subspaces labeled by the groups H, which in this case are the 
unramified tori (split and non-split) in GL2. They are labeled by the unramified double 
covers of X. 

Let us look at the vector space (j7.2ip more closely. The sheaf ^ is supported on the 
locus of rank two bundles which have the form (j7.7p : 

(7.22) ^ kII^ ^ K^^^^ 0. 

The support of ^ consists of bundles V of this form. We will fix a point 00 G X and 
identify rank two bundles V and "\7(n[oo]) for all n (z Z. In other words, we replace 
BunG'2.2 by its quotient by the group Z generated by tensoring rank two vector bundles 
with the line bundle 0([cxd]). 

When we apply our averaging functor ^d,p, we average over the Hecke modifications 
of the bundle V: for each divisor D = ^^riilxi] of degree d we apply all possible Hecke 
modifications corresponding to the representation Sym*^' p at the point Xi, for all i, 
and then take the resulting bundle V and replace it by 'V{—d[oo]). This bundle should 
then again be of the form (|7.22p . Thus, the space (I7.2ip can be interpreted as the 
cohomology of a certain sheaf over the moduli space W^^p of data (V, V, D, 0), where 
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D = Yl^A^i]^ V are rank two vector bundles on X the form ()7.22p . together 

with an embedding 

: V V'(d[oo]), 

such that the quotient 

V'(d[oo])/V^0T,,, 

i 

where are torsion sheaves supported at Xj: 

'^x, ^ Ok,x, © O(2n,;-fc0a;,.' fo'^ some ki = 0,...,ni 

(these correspond to the strata in the affine Grassmannian which he in the closure of 
Gr„-p). 

We need to compute the cohomology of this sheaf and relate it to the cohomologies 
corresponding to the contributions of "^H in the sum on the right hand side of (j7.20p . 
We hope that we could use the methods of [Nl] and |FGVlj to do this. 

7.8. The abelian case. Finally, we consider the case of the group GLi and trivial p. 
We wish to check the statement of Section 17.41 that the cohomology (j7.9p is really the 
geometric incarnation of the sum appearing on the left hand side of ()7.5p . 

Since p is trivial, the symmetric power of the curve X^i decouples from the formulas, 
and we might as well set K = Id, and so Nu = 1 in formula ()7.5p . Note that the adjoint 
representation of GLi is trivial, and so 

g-'^«L(a,ad,.) = g-(f-i)c(s), 

where ({s) is the zeta-function of X. However, since GLi is not simple, we have to 
make an adjustment and replace C,{1) by its residue at s = 1, which we denote by C(l)- 
In other words, we remove the factor corresponding to H^{X, Qe), which is independent 
of a. (One can argue that this factor decouples because the structure sheaf of the full 
derived stack of rank one local systems on X contains the exterior algebra of H^{X, Qi) 
placed in the cohomological degree 1.) 
Thus, we have to compute the sum 

(7.23) Yl I'^'^'^Ci^r' = q-^'-'^aiy' ■ #W ■■ Wf ^ GL,}, 

c 

where a runs over all unramified homomorphisms Wp — > GL\. 

Let ai^i = 1,... ,2g, be the eigenvalues of the Frobenius on H^{X,Qi). We have 
ai = PiQ^^'^, where |/3i| = 1 and for each /3j there is /3j = . Therefore 

7(..-i ^ det(l-g-^Fr,g°(X,Q,)) ^ , q-1 

det{l-q-^FT,H^X,qe)) n'£i(l-«.)' 

On the other hand, by the abelian class field theory, the number of unramified a : 
Wp GLi is equal to the dimension of the space of unramified automorphic forms 
on GLi(A), which is the number of Fg-points of the Jacobian of X. This number is in 
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turn equal to the alternating sum of the traces of the Frobenius of the cohomology of 
the Jacobian, which is A*H^[X,Qi). Hence the answer is 

1=1 

Thus, the sum (I7.23P is equal to q — 1. 

Its geometric incarnation is the vector space 

(7.24) H'{LocGL„0) 

(since we have replaced M.ii^p with the identity functor, we replace the bundle 7d^p with 
the trivial bundle 0). 

It is easy to see that H^iLoccLn 0) is one-dimensional, and W{Locgli-, 0) = for 
all i > 0, so that the total space (I7.24p is one-dimensional, situated in cohomological 
degree 0. This means that the (non-existent) Frobenius should act on it by (7 — 1. Let 
us compare this number with the number appearing on the right hand side. 

First, let us look at the right hand side ()7.15p of the relative geometric trace formula 
(j7.1ip . As above, we replace IK^^p by the identify functor. The resulting vector space is 

(7.25) RHom^(Pi,0(^))(^,M'). 

Here Pic'^(X) is the Picard moduli stack of line bundles on X of degree and ^ = 
P\{Q.i)^ where p : pt ^ Pic'^(X) corresponds to the trivial line bundle. This is also a 
one-dimensional vector space situated in cohomological degree zero. So we indeed have 
an isomorphism between ()7.24p and ()7.25p which is to be expected because in the case 
of G = GLi the categorical Langlands correspondence is a theorem of |LauH [R]. 

Now let us compute the action of the Frobenius on the one-dimensional vector space 
()7.25p . In the case when X is defined over Fg, the Frobenius is well-defined. Since 
the trivial line bundle has the group of automorphisms G^, the trace of the Frobenius 
on ()7.25p is the same as that on (/>'</>!(Q^), where (/> : pt — >■ pt /Gm, which is equal to 
#Gm(IFg) = q — 1. This matches the above calculation for ()7.24p . 

Finally, let us verify formula (j6.12p in the case G = GLi. Since the tangent bun- 
dle to LocGLi is trivial, with the fiber H'^{X,C) = H^{X,0) H^{X,Kx), and 
i7'(LocGLi,0) = C, the right hand side of (l6T2]l is equal to I\*{H^{X,C)), which 
is also isomorphic to the left hand side of (I6.12p . 
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